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ABSTRACT 

The unique factor in the Medical Information Project 
is that wording from ground zero, it undertook to design and put into 
operation a communication system for general medical practitioners 

using an individualized, programmed, audiovisual medium. The 
development of this system involved three general phases. Phase I, 
consisted of: ( 1) obtaining and reviewing literature pertaining to 

medical communication problems; (2) designing a means of sampling, 
drawing the sample, obtaining the physicians* participation; (3) 
laying out the general design for research and development; (4) 
developing and validating the instruments to be used to assess the 
physicians* cognitive and affective reactions; (5) testing and 
selecting the hardware to be used as the communication device; (6) 
developing the programming concepts; (7) developing the production 
process and (8) selecting the content areas and the medical 
consultants for the program topics. Phase II, consisted of (1) 
distributing the hardware to the participating physicians, (2) 
producing and distributing the training program on equipment 
utilization, (3) pre-and post-program questionnaires and interviews 
and (4) producing and distributing the fifteen content programs. 

Phase III consisted of: (1) collecting and processing raw data, (2) 

analyzing the data and (3) writing the final report. (Author/NH) 
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FOREWORD 



The Medical Information Project was conceived 
and created by James D. Finn, Professor of 
Education and Chairman, Department of Instruc- 
tional Technology, University of Southern Cali- 
fornia . From its inception, the Project was 
guided through many difficult and sometimes 
frustrating days by his experienced and dynamic 
leadership. The death of Dr. Finn in April, 
1969, was a severe blow to the Project and to 
those who worked with him. We cannot help 
believing that some of the delays in completion 
and other problems encountered would have been 
obviated by the mere presence and leadership of 
Dr. Finn. That the Project has been completed 
is mute testimony to the expertness and vision 
in design of the study provided by Dr. Finn. 

The Final Report of the Medical Information Pro- 
ject is presented with the knowledge and regret 
that it lacks the personal touch and brilliant 
style for which Jim Finn was so well known. 
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INTRODUCTION 



The problem of the so-called explosion of knowledge has become the sub- 
ject of much concern and study In recent years. As Hoban (1967) pointed out, 
there Is a communications gap between scholars and practicing members of 
professions who are consumers but not producers of scholarship and research. 
This problem has long been particularly acute in the medical profession. New 
information, concepts, theories, even whole new fields of operation continue 
to invade the field of medicine with increasing force. A case in point is the 
practicing physician who faces the difficult and complex problem of attempt- 
ing to keep abreast of the advancement of medical knowledge. The importance 
of continuing education for the physician was discussed by Abelson (1965): 

One consequence of the large-scale activity in research is the 
obsolescence, at least to some degree, of all scientists, en- 
gineers and physicians. The problem is not new, but the rate 
of obsolescence has increased, while the traditional means of 
meeting it have become less effective. 

Dryer (1962), proposing lifetime learning for physicians, cited the Report 

of the President's Conference on Heart Disease and Cancer: 

. . .the best medical education and training can become obsolete 
in 5 years unless the physician makes a very determined effort to 
continue his education. . . .[p. 676] 

Efforts embracing many techniques of bringing information to segments 
of the medical profession have been and are being attempted. Medical schools 
now have departments of "continuing education"; seminars, lectures, Journals, 
television, and many other information channels are used in an effort to deal 
with the information flow. Yet the gap between theory and application of 
medical knowledge remains . As Clute (1963) said: 
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Suppose tnus. this vital link were broekn when a physician passed 
from his postgraduate training into practice, i.e., that there 
were no continuing education. In times past, it would have made 
little practical difference, because major advances were infrequent; 
but the breaking of this vital link in recent years would have meant 
that physicians who are at present over fifty years of age would be 
practicing the medicine of more than twenty-five years ago, when 
there ws&e no sulphonamides , no antibiotics, no antihistamines , 
no ACTh, or cortisone, and no antihypertensive drugs [p. 448]. 

The explosion of knowledge becomes more evident when one views a tra- 
ditional method of acquiring professional information — the medical journal. 
Roney (1962) described it thus: 

The problem of quantity is well documented. . .In the field of 
medicine, the world literature includes 4,000 to 5,000 journals 
which publish 220,000 to 250,000 articles per year. It has been 
estimated that for a physiologist to read all the physiological 
literature published in 1960, it would require approximately 
three and one-half years, providing he read a page every two 
minutes during eight hours of each day [p. 564]. 

This multitude of literature is of questionable value to the busy practiti- 
oner who has little time to even pick out the materials that are valuable to his 
field, much less to read them. This has encouraged the development of di- 
gests and abstract services. It has been estimated (Roney, 1962) that even 
the number of digests and abstracts has increased by a factor of ten every 
fifty years, and that for every 300 Journals there is one abstract journal. 

The rapid expansion of medical knowledge and the inefficiency of tradi- 
tional methods of keeping abreast of such knowledge is compounded by the 
fact that most doctors are busier than ever before. Cahal (1962) stated that 
the general practitioner sees an average of 190 patients in a typical five and 
one-half day week, during which he works about 60 hours per week. It is ob- 
vious that the time the physicians can devote to postgraduate education is 
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limited. As Abel son (1965) stated: 

The bottleneck in utilization of knowledge is not a shortage of 
publications or inadequate information retrieval . The lag occurs 
in the step between the pile of books on a man's desk and the 
transfer of that information to his mind. We need to devote 
much more energy, to determining what is significant and then 
conveying it in concentrated form . 

Attempts have been made to partially digest available information and to 
bring the physician closer to the information he needs by applying technology 
to medical education. Closed- circuit television, two-way radio, and tapes 
such as those produced by Audio-Digest are evidence that the medical estab- 
lishment has recognized that there is. a technology of instruction. One of the 
specific areas in which the technology is being developed is individualized 
instruction , presented via "teaching machines" ranging in sophistication 
from simple cardboard frames to computers. Individualized audiovisual instruc- 
tion, particularly for training in procedural skills, is regarded by many educa- 
tors to be more effective than standard classroom instruction because it allows 
the student to repeat a lesson as often as necessary and to choose the time and 
perhaps the place of instruction. This form of instruction is now being used 
successfully in hospital orientation, in-service training programs, and stu- 
dent resident training in specialist areas. 

In addition, there was a firm basis for suggesting an audiovisual approach 
to communication. Many studies have indicated that authoritative commentary, 

t 

careful programming, and color, where color is an important factor in the com- 
munication (e.g. , in diagnosis) are significant in the teaching-learning pro- 
cess, assuming a good content design. 

The Medical Information Project devised a system, based on instructional 
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technology, to attempt to solve the twin problems of growth of information and 
lack of time. It was assumed that (l)the general practitioner is overwhelmed 
by the mass of printed materials which he probably should read, (2) he is too 
busy to participate in other types of continuing education (such as postgradu- 
ate courses) , and (3) he may be deterred by other factors such as time in- 
volved, inconvenience of scheduling, travel and cost from many other types 
of postgraduate education. 

It was concluded (1) that the system must be conceived as serving a 
busy man, the individual physician. (The device, which could be placed in 
the physician's office or home, should be capable of interruption and easy 
resumption if necessary); (2) that the content must be relevant to his indivi- 
dual practice needs; (3) that the key to better communication of such content 
was the design of the materials themselves (every effort would have to be 
made to use all possible knowledge and experience in designing the materials 
so that they would communicate efficiently and lead to further learning); (4) 
that the system must be as easy to operate as possible, requiring a minimum 
of effort on the part of the physician. The individual device must be rugged; 
the materials must arrive on time and in good condition; and the reporting of 
information necessary for the purposes of the study should require a minimum 
of the doctor's time. 

What is reported here, then, is a study of the effectiveness of one ap- 
proach which was developed under the restraints of the above limitations. 
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Figure 1 

School of Education University of Southern California 





Figure 2 

School of Medicine University of Southern California 
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Figure 3 

Los Angeles County - University of Southern California 
Medical Center 





Figure 4 

Norris Library U.S.C. School of Medicine 
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Figure 5 

Principles Involved in the Medical Information Project 
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CONDUCT OF THE STUDY 



The Study 

The Medical Information Project, a joint research and development pro- 
ject by the School of Medicine and the School of Education of the University 
of Southern California, was an attempt to design and test an audiovisual 
communication system for 100 general practitioners in eleven western states. 
In the three-and-one-half-year contract period, MIP explored the communica- 
tion potential of individualized audiovisual programs on fifteen medical topics 
presented by a device synchronizing projected still pictures and a recorded 
narration. 

The unique factor in the Medical Information Project is that working from 
ground zero, it undertook to design and put into operation a communication 
system for general medical practitioners using an individualized, programmed, 
audiovisual medium. The development of this system involved three general 
phases: the Preliminary or Initial Planning Phase, the Production and Program 
Delivery Phase, and the Evaluation Phase. In this connection, Brickell (1964) 
has stated, 

. . .it is one thing to design or invent a new way of teaching, it 
is another to find out whether the invention is any good, and it 
is still another to demonstrate it for the purpose of persuading 
others to adopt it. That is to say, the design , evaluation , and 
dissemination of the innovations are three distinctly different, 
irreconcilable processes [p. 493]. 

This project recognized this same distinction, but does not hold that 
the three processes have no interrelationships. Rather, a distinction is made 
between evaluation and feasibility on the one hand and rigorous, controlled 
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measurement on the other. Because the system was designed with no 
precedents upon which to rely , the project emphasis was on develop- 
ment and field testing. 

The Objectives 

The first objective was to develop, and test the feasibility of, 
a technology of Individualized communication for the general practitioner. 
Emphasis was placed on the design of materials and their relationship 
to the communication needs and behavior of the general practitioner. 

"Feasibility" in this context actually constituted a second 
objective. The technology utilized during the course of the project was 
selected with an eye to accomplishing the objectives with as low a 
cost as compatible with the defined objectives of the communication. 

A third general objective related to the development and study of 
the process of individual communication as a system . In other words, 
an important objective was to develop a system of communication that 
was operable and generallzable to other situations within medicine, and 
to define and describe the system in sufficient detail so that it might 
be used by other medical communicators. 

Finally, an effort was made to develop new ways in which such a 
technological approach could be improved and also to open the way for 
highly controlled studies in the field of medical communication as related 
to communication technology. 

Pe rsonnel 

The Medical Information Project was directed by Dr. James D. Finn 
of the School of Education from October, 1966 until his death in April, 1969. 
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Dr. Stephen Abrahamson of the School of Medicine was the Associate Director 
and became Director upon Dr. Finn's death. Mrs. Diana Caput, however, was 
instrumental in coordinating the completion of the study. The staff consisted 
of four segments: (1) the programming/production staff, (2) the research staff, 
(3) the medical staff, and (4) the administrative staff. 

The production/programming staff consisted of a producer/writer, an 
artist, and a photographer. Writer/researchers were used on a program-by- 
program basis. The research staff consisted of the research director, who 
also worked on programming, and a research assistant. The medical staff con- 
sisted of a consultant selected from the School of Medicine faculty for each 
program by Associate Dean Phil Manning. The administrative staff consisted 
of an administrative assistant, a secretary, and some part-time clerical help. 
Advisory Committee 

Dean Manning established the overall monitor of the project — the Medi- 
cal Information Project Advisory Committee. This committee, provided for 
in the contract, (1) prescribed the areas of medicine to be covered in each 
program, (2) suggested possible consultants, (3) gave final approval on pro- 
grams before they were mailed, and (4) monitored the general conduct of the 
project through staff reports and discussion. 

The original Advisory Committee consisted of 

Phil R. Manning, M.D. 

Associate Dean, Postgraduate Medicine 

USC School of Medicine 

Ralph Bennett, M.D. 

Past President 

California Academy of General Practice 
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Norman Shrifter, M.D. 

Associate Clinical Professor of Medicine 
USC School of Medicine 

J. Samuel Denson, M.D. 

Professor of Surgery 
Chairman of Anesthesiology 
USC School of Medicine 

Donald W. Petit, M.D. 

Associate Professor of Medicine 
USC School of Medicine 

Leonard H. Schwartz, M.D. 

Assistant Medical Director and Director, 

Out-Patient Department 
LAC-USC Medical Center 

Due to illness and a change of assignment, Drs. Bennett and Petit were re- 
placed by: 



Dudley M. Cobb, M.D. 

President 

California Academy of General Practice 

Richard W. Opfell, M.D. 

Continuing Medical Education Program 
California Medical Association. 

Time Schedule 

The overall time of the project was from October 1, 1966 to March 31, 
1970 — a period of three and one-half years. The timing was phased approxi- 
mately as follows: 

Phase 1 Initial Preparation 

Phase I covered the period from October 1, 1966, to April 15, 1967. The 
original plan of six months for these activities was delayed nine months be- 
cause of difficulties in obtaining clearance from the Bureau of the Budget for 
the research instruments. The preliminary phase consisted of (1) obtaining 
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and reviewing literature pertaining to medical communication problems; (2) de- 
signing a means of sampling, drawing the sample, obtaining the physicians' 
participation; (3) laying out the general design for research and development; 

(4) developing and validating the instruments to be used to assess the physi- 
cians' cognitive and affective reactions; (5) testing and selecting the hard- 
ware to be used as the communication device for the project; (6) developing 
the programming concepts; (7) developing the production process, including 
studies of formats, etc.; and (8) selecting the content areas and the medical 
consultants for the program topics. 

Phase II Production/Proqram Delivery 

While production began in April, 1967, the experimental period itself 
ran from April, 1968 through August, 1969. The major activities involved in 
this phase were (1) distributing the hardware to the participating physicians; 

(2) producing and distributing the introductory training program on equipment 
utilization; (3) administering pre- and post-program questionnaires and inter- 
views; and (4) producing and distributing the fifteen content programs. 

Phase III Evaluation and Report Writing 

The evaluation activities occupied the final eight months of the project 
from August, 1969 through March, 1970. These consisted of (1) collecting 
and processing raw data, (2) analyzing the data, and (3) writing the final 
report . 

Obviously there was some overlap in these phases. For example, although 
the major emphasis on evaluation was during the last several months of the 
project, data were being collected and tabulated from the onset of the production/ 
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program delivery phase with the mailing of the first questionnaires . Figure 6 
PERT Chart developed at the beginning of the project, shows some of these 
interrelationships of activities during various phases of the project. 
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PHASE I: INITIAL PREPARATION 
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PHASE I: INITIAL PREPARATION 



Review of the Literature 

During the initial planning period and continuing through most of the 
production phase, an extensive literature search was undertaken. This pro- 
vided information relating to the communications behavior of physicians, 
factors that influence this behavior, the nature of the target audience, and 
their needs and problems in acquiring professional information. Other attempts 
to use media in medical education were also examined. This preliminary 
examination provided a basis for drawing general guidelines for the design 
of the materials to be produced during the production phase. A report on this 
literature search was published by the Project as Research Memorandum Number 
1: Interim Report on the Search Bibliography and Document Collection, October 
1967 . 

Communication of important research information is often not available 
in the literature due to time lag or, as several students of information flow have 
noted, is of the type not ordinarily reported in the formal literature. There- 
fore, personal contact was made with three groups of people who were knowledge 
able in this area: (1) those with an academic interest in the problems of 
medical communication, (2) representatives of associations, such as the 
American Medical Association and the American Academy of General Practice, 
who have communication know-how and an interest in communications, and 
(3) communication experts in the pharmaceutical industry. 
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Testing and Selection of Hardware 

A general concept of the system design was an audiovisual device which 
could be placed in the physician's office and which would tolerate interruption. 
At the outset of the project, an extensive evaluation and testing of existing 
hardware was conducted to select a machine that would most nearly meet the 
ideal requirements of programming and use specifications of the Project, and 
which could be purchased in a lot of 100 at a reasonable price. 

The display devices available combined 2"x2" slides with cartridge- 
loaded tapes; conventional filmstrips with disc recordings; cartridge-loaded 
8mm or Super 8mm sound-on-film with stop-frame capability; or a combina- 
tion similar to these. These all have certain common characteristics, the 
most significant being the use of rear projection through a relatively small 
ground-glass or Fresnel type screen. 

Based on the programming concepts selected, on a thorough review of 
the literature pertaining to the target audience, and on discussions with experts 
in medical communication, a set of criteria was established to evaluate these 
machines. Other requirements were added when the operational and economic 
factors became apparent as the project developed. These were (1) reliabi- 
lity, (2) safety, (3) ease of operation, (4) low cost production of software, 

(5) availability and cost of manufacturing, distribution, and servicing. A set 
of specifications was drawn up, the search was instituted, and twenty-seven 
(27) machines were obtained and evaluated. A report on the testing and 
selection of these machines appears as Research Memorandum Number 2 : 

An Analysis of Audiovisual Machines for Individual Program Presentation, Ma y, 
1967. ' ' 
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The display device adopted for use by the Project was the Hoffman Mark 
IV Audiovisual Projector, slightly modified in its programming function. This 
particular device has a screen approximately 5 'lx 8" and combines the visual of 
a filmstrip encapsulated in a sprocketed plastic holder (fourteen 35mm frames 
per strip) which is inserted into the unit and automatically engages the trans- 
port mechanism, and with audio provided by a seven inch record which is in- 
serted into the unit and is automatically positioned on the turn-table. Each 
side of the record provides up to 6-1/2 minutes of playing time at 33-1/3 rpm. 
The Projector has two modes of operation: Automatic Advance and Automatic 
Stop. The device allows for a rudimentary form of programming in that the pro- 
gram can be automatically interrupted at any time to request an overt response 
to a question. The program is then resumed when the re-start button is pushed. 
Production Planning 

After the physical set-up for the production unit had been developed to 
the point where it was operable, a work-training program was begun. Various 
samples of medical and audiovisual materials were obtained and reviewed by 
the production staff. Medical illustration books and samples of current medical 
illustration were provided for study. Tests were run on the equipment. 

Following this, a training assignment was made to produce a slide-tape 
presentation on the Medical Information Project. In the production of this pro- 
gram, format studies were conducted and tested for labeling, shape, size, and 
composition. Cost analyses of record production, filmstrip printing, etc. , 
were conducted to develop the most economical procedure for producing and 
distributing the programs . 




29 



IS 



I 

l 

1: 

r 

I. 



y 

y 

y 

y 

ii 



y 




Operationally, a multitude of details, techniques, and processes, had 
to be developed in order to design, produce, and deliver the programs. One 
example is illustrative. Albums were developed to include the encapsulated 
filmstrips, the recordings, the program booklet, and other printed materials. 

The album cover was designed; a mock-up tried out revised, and then re- 
produced in large quantities sufficient to meet the needs of the project. 

The programming/production process also had to have a feedback loop 
of physician-reactions in order to continuously monitor and improve the pro- 
cess. This necessitated setting up a somewhat elaborate system of compiling 
program evaluation information and feeding it to the production group and the 
Advisory Committee. 

Programming Concepts 

The programming concepts were developed at the outset of the Project. 

A few years ago, programmed instruction, according to most of its proponents, 
had to be developed according to one of several defined and opposing theories. 
The two most often suggested as opposites were (1) the small-step behavior- 
shaping, successive-approximation, constructed-response, no-mistake approach 
of B. F. Skinner and (2) the larger-step, branching if a mistake is made, 
multiple-choiC'i response then associated with Norman Crowder. Sometimes 
Skinner and Pressey were set in opposition with (and this is highly over- 
simplified) multiple -choice versus constructed-response, mistakes allowed 
versus always correct, etc. 

Recently, however, the entire programmed instruction movement seems 
to have broadened its base and become more eclectic in its approach. Study 




of one authoritative publication (Lange, 1967) reveals that rather than viewing 
programming as related to a particular psychology theory, it is now thought of 
as an engineering process involving (1) a clear statement of objectives, pre- 
ferably in behavioral form, (2) the design of a program built to achieve these 
objectives, using any combination of theories and media, (3) the validation of 
this program on individual subjects and, later, the field-testing of it on popula- 
tions of students with subsequent revision, and (4) release of the program for 
instructional purposes complete with performance data derived from the testing. 

The general strategy for preparing the programs for the Medical Informa- 
tion Project is in this new eclectic tradition, but not derived from it. Essential- 
ly, it is derived from S. L. Pressey's theories of the last few years and is best 
referred to in Pressey’s nomenclature as auto-elucldatlve. " Following a dis- 
cussion of his general theory, Pressey(1963) referred directly to program con- 
struction in the following words: 

. . .initial presentation of what is to be learned will be in field trip, 
demonstration or experiment, or most commonly a substantial unit 
like an inclusive textbook chapter, not all mixed up with autoinstruc- 
tion. The "autodiscussion" would follow and its function would be. . . 
to enhance the clarity and stability of cognitive structure by correct- 
ing misconceptions, and deferring the instruction of new matter until 
there had been such clarification and elucidation. 

In difficult matter such as a science text or industrial or military 
training manuals bits of auto-instruction may be needed more 
frequently; each step in the solution of a difficult problem may 
need such auto-elucidation. . .if the auto-instruction is thus to 
follow presentation of what is to be learned, then. . .it will deal 
only with issues which need further clarification or emphasis. 

Such adjunct auto-elucidation will not cover everything, may jump 
from one point to another or even back and forth [p. 3] . (Under- 
lining in the first sentence of the second paragraph added for emphasis.) 

This approach, modified somewhat by what has been learned about audio- 
visual presentations, was employed in the design of the Project materials. 
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There Is a gap between programming theory and actual programming, however, 
and this Is particularly true when medical expert, writer, producer, photog- 
rapher, artist, narrator, and program-frame-writer must be coordinated in a 
consciously creative effort. A strategy had to be supported by precise tactics. 

The early effort at doing this Is contained in Research Memorandum Number 3 : 

Strategy and Tactics for Program Presentation, May 1967. 

Gener al De sign of the Study (E xperimental Phase ) . 

The design of the experimental phase of the study entailed drawing a 
random sample of 100 general practitioners in eleven western states who would 
participate in the study. A control group of general practitioners was drawn on 
the same basis . 

Th^ - experimental physicians who participated in the study were asked to 
complete pre-and post-program questionnaires covering their usual methods 
of obtaining medical information, such as medical meetings, staff meetings, 
journals, postgraduate courses, as well as other factors that might be related 
to their communications behavior, sucn as distance from medical schools. In 
addition a number of the participants were selected for interview before and 
after the program delivery phase. During the experimental phase, fifteen con- 
tent programs were mailed to the participants at the rate of one per month (approxi- 
mately). After viewing the programs, each physician was asked to complete 
an evaluation form and a short content test on that program. 

% 

The control group of physicians was asked to complete the same pre- and 
post-program questionnaires and the content tests. 

The design for the evaluation of the system of individualized audiovisual 



communication for the continuing education of general practitioners is presented 
in the paradigm below. Several symbols, most of which have been adapted 
from those used by Campbell and Stanley (1966), are identified here: 




E — the experimental group^ 

C — the control group 

R — randomization that took place in the selection of 
subjects for the experimental and control groups 

— initial observations obtained from use of pretest (pre- 
program questionnaire) 

— terminal observations obtained from use of posttest 
(post-program questionnaire) 

X — treatment applied 

N — number of subjects employed in a given group 



The arrangement of symbols from left to right within the paradigm cor- 



responds to the temporal order in which the major steps of the experiment 
were undertaken. Thus, the design for studying change in communication be- 



havior may be portrayed as follows: 



E: R 0, X 0 
1 2 

C:R0„ 0, 

3 4 



Practical circumstances, such as time and cost requirements, argued for this 
type of design rather than for a four-group design. 

In evaluating content learning for each of the specific programs , the post- 
test-only control group design was used. There is evidence that subjects are 
sensitized by pre-tests, and hence, when testing for factual knowledge, post- 
test scores are almost always enhanced as a result of the pre-testing. In a 
study reported by Edling (1964), it was found that the differences between the 

control and film group in the before after design were not statistically signifi- 
cant, while differences between the control and film group in the after-only 
design were significant beyond the 1 per cent level. It was inferred from this 




that the pretest, in interacting with the communication, masked its effective- 



ness . 



Consequently, the symbols 0, and 0 in the paradigm for evaluating 

the effectiveness of the programs with respect to content learning refer to 

observations obtained from the use of post-test only for the experimental and 

control groups. The design may be portrayed as follows: 

E; R X Oj 

C: R 0 0 
2 

Some Cautions on the Design 

In a research and development study such as this one, which may possess 
a great deal of novelty or special appeal to the participants, one needs to be 
aware of the possible operation of certain psychological factors that cannot be 
completely controlled by even the most sophisticated experimental designs. 
Among the factors that may be operative specific attention needs to be dir- 
ected toward the following: 

1. The familiar "Hawthorne effect" in the experimental groups 
that may be associated with marked changes of a temporary 
nature in their responses because of the novelty and prestige 
that participation in the study affords . The lengthy time 
period of the study mitigates against this possibility. 

2. The, tendency during the interview to please or to impress 

the experimenters — the risk of the introduction of the response 
set of acquiscence and social desirability. The physician's 
responses to the questionnaires, evaluation forms and con- 
tent tests were apparently anonymous, which should have 
lessened this tendency with respect to these forms. (These 
forms were coded for identification purposes.) 

3. The threat to self-esteem that may lead some physicians to 
do a great deal of extra work or extra study of information 
sources so that they can honestly show changes in behavior 
that project the image of increased competencies in medical 
knowledge and practice. 
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4. The occurrence of communication among participants in the experi- 
mental and control groups concerning the objectives and outcomes 
they anticipate for the study — a tendency for reactive effects 

of the experimental arrangements (Campbell & Stanley, 

1966, p. 6) to invalidate the outcomes. This is highly un- 
likely in this case due to the distribution of the physicians 
throughout the eleven western states . 

5 . The presence of marked differences in the frustration tolerance 
and degree of conscientious perserverence on the part of the 
physicians in keeping detailed records and accurate informa- 
tion on the extent and type of use which they make of communica- 
tion materials — differences in temperament or styles of work 

that may lead to errors in recording their responses to items 
in the questionnaire during the post-test experience . 

Statistical Treatment 



The methods employed throughout this study for testing the statistical 
significance of the observations to rule out the possibility of chance happen- 
ings consisted of standard procedures. These included the familiar t-test 
for significance of difference between two means, McNemar's Chi Square test 
for correlated frequencies or proportions, the Chi Square test for independent 
groups, and other non-parametric statistical tests. 

The experimental and control group mean scores for each of the content 
tests were evaluated through the t-test for significance of difference between 
two independent means. Results stated as statistically significant were such 
that the probability of their occurrence by chance alone was always 5 per cent 
or less. In some cases, the exact probability was stated. 

Research Instruments 

Several months were spent in carefully designing appropriate instruments 
which would elicit the information required to evaluate various aspects of the 
project. These materials included (1) pre- and post-program questionnaires, 
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(2) an interview guide, (3) program evaluation forms, and (4) content tests. 

All were systematically pretested by personal interview with a panel of select- 
ed general practitioners in the Los Angeles area and revisions made based 
upon their comments prior to utilization in the study. 

The MIP research instruments may have the distinction of being the most 
thoroughly tested and analyzed forms utilized in a federally funded project. 

They were prepared, tested, revised, tested again by personal interview, and 
again revised . They were then submitted to the U . S . Public Health Service 
for approval by the Bureau of the Budget. The instruments were reviewed by 
personnel in the Behavioral Science Section of the Division of Community 
Health Services. Based on their recommendations, the instruments were again 
revised (which in some instances entailed reconciling two differing analyses) , 
tested, and then resubmitted for clearance. Personnel within the Bureau of 
the Budget made some further suggestions for revisions, and after some dis- 
cussions by telephone, the revisions were made in order to expedite clear- 
ance by that office. Approval of the instruments came nine months after they 
were submitted. 

To achieve anonymity of responses and at the same time to keep record 
of who responded and who did not to each of the research instruments, a 
number code was assigned to each physician in the control and experimental 
samples, and affixed to each form. 

A brief description of each of the instruments is presented in the follow- 
ing paragraphs, and a copy of each is included in the Appendix of the report. 

Questionnaires . The pre- and post-program questionnaires were designed 





to serve as the major source of data for investigating any changes in communi- 
cation behavior in the physicians. These instruments provided data on the 
activities that the general practitioner exhibits in relation to his professional 
duties and particularly to his efforts to keep abreast of the expanding universe 
of medical knowledge and of modifications and innovations in clinical procedures . 
The questionnaires also elicited information on other factors that might be re- 
levant to, or influence, the communications behavior of physicians, such as 
distance from medical schools, medical practice arrangements, number of 
hours worked. The final version that was administered to both the experimental 
and control groups of physicians prior to and again at the conclusion of the 
program delivery phase of the project ccnsistedof 25 items, printed in pamphlet 
form, requiring about 30 minutes for completion. 

The pre-program questionnaires were mailed with a covering letter to both 
groups of physicians on April 5, 1968. Return envelopes were printed, con- 
spicuously stamped, and marked First Class. This special attention with the 
return envelopes was intended (a) to facilitate the mailing of the response and 
(b) to reinforce the professional predisposition to respond. Three weeks later 
a follow-up letter was sent to all who had not responded. No further attempt 
was made to solicit information from members of the control group who did not 
respond to the first or second request. The experimental group, however, who 
had agreed to participate in the study, were contacted by telephone if they had 
not responded to the first and second letters. 

The post - program questionnaires were mailed to the control group of 
physicians on July 14 and to the experimental group on July 21, 1969. Follow up 




letters were sent to non-responding control group physicians on August 8 and 
to the experimental group on August 14, 1969. A massive telephone campaign 
was begun August 25, 1969 to urge the physicians' help in filling out the 
questionnaires as soon as possible. A postcard reminder was sent to non- 
responders on September 3, 1969. 

Interview Schedule . To make procedures uniform and to provide a basis 
for gathering appropriate information, an interview guide for use by project 
field personnel was developed. As a validation device for the questionnaires 
and as a means for acquiring in-depth information from physicians regarding 
their needs for information and their ways of obtaining it and their attitudes and 
motivations relating to their communications patterns, intervie vjs were carried 
out both prior and subsequent to the series of fifteen programs. Pre-program 
interviews, conducted with about 60% of the participating physicians provided 
information which served as a guide in the design of the specific program 
materials to be used during the course of the project. Post- program interviews 
were designed to provide information which would serve to guide the improve- 
ment of such a system of individualized, audio-visual, programmed instruc- 
tion. 

Program Booklets . Accompanying each of the fifteen audio-visual pro- 
grams was a program booklet which included (1) program questions to be used 
in conjunction with the audio-visual program itself, (2) any tables of informa- 
tion that the general practitioner might want to keep for easy reference, (3) 
a program evaluation form, and (4) a content test. The booklet was constructed 
in such a way that the middle pages, consisting of the evaluation form and 



content test, could be easily removed from the booklet and returned to the pro- 
ject in a prepaid envelope. 

Program Evaluation Form . The program evaluation forms for each of the 
fifteen programs were designed to furnish information on the attractiveness and 
acceptability of each of the- instructional units . The data from these forms pro- 
vided feedback to the production team to guide in the design of future programs. 
The final form consisted of 11 items eliciting information regarding types of 
subject — matter content, design features, etc. which might best be included 
in this mode of presentation, and how physicians view other sources of informa- 
tion in terms of their perceived needs . 

Content Tests . The content tests were designed to provide information 
not only on how well the physician had learned, but also on how well the 
material in the program had been presented. One of the cardinal concepts of 
programmed instruction is that if the student fails, the program also fails. 

Test questions, while measuring content knowledge, also tested how closely 
the MIP programs met the behavioral objectives. These tests consisted, on 
the average, of ten objective (true-false, multiple -choice) or short answer 
items, requiring about five minutes to complete. These tests were also admini- 
stered to the physicians in the control group, and mailed with the post-question- 
naire . 

The Sample and Sampling Procedure 

It was very important to the Medical Information Project study that the 
physicians be representative of general practitioners in the eleven western 
states. For this reason, the sampling process is described in detail on the 




following pages . 



Defining the Population. 

The primary and most useful source for determining the population from 
which the sample was drawn was the 1965 AMA. Directory (23rd Edition). This 
was the latest edition available and was used even though the work on sampling 
began almost a year later (November, 1966). The 1966-67 AAGP Directory and 
lists provided by the state medical societies were also used for reference . The 
AAGP Directory was not heavily relied upon because it would have introduced 
a bias since only about 33% of the total GP population, nationwide, are members 
of the American Academy of General Practice, and the AAGP prequires its mem- 
bers to engage in a minimum number of hours of postgraduate education. 
Definitions and As sumptions . 

In establishing the population from which the sample was to be drawn, it 
was necessary to consider a definition of general practice. As Peterson (1956) 



An attempt to define or describe general practice involves study 
of it under varying circumstances. It may change from town to 
city, with the age of the doctor, with sea|on of the year, with the 
number of partners or aides, with the size of the physician's in- 
come and with many other factors . It would be necessary to know 
something about the doctor and his training, his patients and their 
diseases , his facilities , the extent of his work and any limitation 
imposed upon his practice [p. 6]. 

In this same vein, Wolf (1965) has stated: 

There are a great number of general practitioners in the U.S. , but 
to. assume that they are all the same is, of course foolhardy. Cer- 
tainly, general practitioners are people with individual differences 
related to their own personal background, medical training and other 
factors . This is a large country with great differences in financial 
assets of communities, geography, population, and the consequent 
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health problems with which doctors must deal. . .Similarly, the 
definitions of a general practitioner is variable in terms of what 
he does (not what he is supposed to do). Although most general 
practitioners are less formally educated than specialists , it is 
not what they have been trained to do but what they must do to meet 
public demand which is important [pp. 737.-738]. 

These comments illustrate the difficulty which faces one who tries to 

arrive at an adequate description of a general practitioner. This appears to 

be a period of transition within the medical profession as exemplified by the 

numerous attempts to define the generalist and his function, the question of 

whether general practice is on the decline or increase, and the whole issue 

of "family" practice versus "general" practice. There is also the question 

of whether or not some specialists (especially pediatricians and internists) 

are, in fact, practicing general medicine. If so, they may soon face some 

of the same problems the general practitioner faces in keeping abreast of 

medical knowledge . 

This study utilized the traditional description of a general practitioner 
as a man who had no specialty training and did not limit his practice to a sin- 
gle field. There are many who fall between this group and those who would 
definitely be described as specialists by virtue of their training, board 
certification, and limitation of practice. In deciding which of these would 
be included in our population of general practitioners, the main emphasis was 
placed on the scope of their practices. Those who may have had some speci- 
alty training (but were not board certified and were still listed as GP's) and 
who did not limit their practices to a specialty field were included. Those 
who may have had very little further training in a specialty field but who did 






limit their practices (e.g. , a physician who practiced only obstetrics) were 
excluded. Physicians in this latter group were not considered specialists. 

They were excluded because it was the main purpose of this project to com- 
municate a wide range of information to those physicians who practiced a 
broad range of medicine. 

Universe and Frame . 

Given the working definition of a general practitioner, a number of factors 
of potential importance were considered before selecting the sample. 

(1) Age. General practitioners who were sixty-five years of. age and 
older at the start of the project were deleted from the population. This was 
done because it was anticipated that doctors in this age group would be 
more likely to (a) be near retirement and/or (b) have limited practices . In 
a small sample such as this, it was necessary to minimize the possibility 
of "dropouts" due to retirement. It was also assumed that physicians with 
limited practices would be less likely to be as concerned about the informa- 
tion problem as those who are maintaining full-time practices. 

(2) Type of Practice. Those physicians who were in institutional 
practices, in veterans' hospitals, in the armed services, in medical schools, 
or in training programs were deleted from the population. 

(3) Area of Practice . Physicians who specified two or more specialty 
listings were deleted from the population. 

Thus, the population was defined as general practitioners in the eleven 
(11) western states who (1) were in full time private practice, (2) were under 
sixty-five years of age and (3) did not limit their practice to a single area 
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of interest. 

In this way the universe of 12,440 was reduced to a frame of 9,605 gen- 
eral practitioners as shown in Table 1 . 

TA0LE 1 



Distribution of Population of General Practitioners in Eleven (11) Western States 



State 


Universe 


Frame 


Arizona 


520 


401 


California 


7,955 


5,928 


Colorado 


701 


552 


Idaho 


276 


239 


Montana 


261 


236 


Nevada 


125 


95 


New Mexico 


225 


176 


Oregon 


719 


591 


Utah 


304 


248 


Washington 


1,198 


1,001 


Wyoming 


156 


138 


TOTALS 


12,440 


9,605 



Size and Selection of Sample 

The size of the sample was set in the contract as one hundred general 
practitioners in the eleven (11) western states. Since this represents only 
about one percent (1%) of the eligible general practitioner population, great 
care was taken in developing the sampling procedures. A disproportionate 
stratified random cample was drawn rather than a simple random sample or a 
proportionate stratified random sample. The reasons for this are discussed 
in the next section. 

Stratification Procedures 

The decision to stratify disproportionately by state was based on three 
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criteria. First, it was necessary to insure that general practitioners in each 
of the eleven states would be represented. Table 2 shows the problem that 
would arise if only a simple random sample were used. California accounts 
for sixty-one per cent (61%) of the general practitioners in the population. It 
is highly likely that the states with a lesser population of general practitioners 
would not be adequately represented. 

TABLE 2 



Percentage Distribution of Population of General Practitioners in Eleven Western 
States 



State 


GP Population 


% of Total 


Arisona 


401 


4% 


California 


5,928 


61% 


Colorado 


552 


6% 


Idaho 


239 


3% 


Montana 


236 


3% 


Nevada 


95 


1% 


New Mexico 


176 


2% 


Oregon 


591 


6% 


Utah 


248 


3% 


Washington 


1,001 


10% 


Wyoming 


138 


1% 


TOTALS 


9,605 


100% 



Second, it was considered desirable to have general practitioners from 
each state adequately represented because of the varying availability in each 
state for continuing education activities and the concomitant effect this might 
have on communications needs and behavior. For example, Idaho, Montana, 
and Wyoming do not have medical schools. This would limit the ease with 
which general practitioners within these states could attend postgraduate 
courses. On ' • other hand, the University of Utah College of Medicine has 

0 
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done a great deal of work with continuing education programs via television 
(Michael, 1963; Castle, 1963) on a statewide basis. In California and Wash- 
ington numerous kinds of continuing educational programs (e.g. , postgraduate 
courses, medical television, tapes) are available. Thus, it was an important 
part of the study to reach those doctors who had limited as well as ready access 
to continuing education in their state. 

Another factor which was considered in the decision to stratify disprop- 
ortionately was the location of the doctors' practices with regard to an urban- 
rural classification. In defining "urban" and "rural", the classification used 
was the one developed by the U.S. Public Health Service, on the basis of the 
I960 Census of the Population projected to 1962 by Sales Management Incorpor- 
ated. The five county groups (listed by degree of urbanization) are (Theodore 



& Sutter, 1966): 

Group 1: greater metropolitan — 109 counties in SMSA's with 1,000,000 

or more inhabitants . 

Group 2: lesser metropolitan — 301 counties in SMSA's with 50,000 

to 1,000,000 inhabitants. 

Group 3: adjacent — 889 counties contiguous to metropolitan areas 

(population in such counties ranges from 500 to 508,500 
inhabitants) . 

Group 4: isolated semi-rural — 1,024 counties containing at least 

one incorporated place with 2,500 or more inhabitants. 

Group 5: isolated rural — 758 counties not included in the above four 
groups . 

Groups 1,2, and 3 are considered urban; 4 and 5 are rural.) 

This classification is a further development of the metropolitan area 

concept of integral economic and social units with distinct population 

centers. The first two groups are determined by the SMSA's established 

by the Bureau of the Budget. [Each Standard Metropolitan Statistical 
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Area consists of the county in which the central city is located and the 
adjoining counties, which along with the primary county comprise an 
integral economic and social unit.] The third group consists of counties 
which border the S MSA's . . .Finally, the fourth and fifth groups comprise 
the rural areas of the country [pp. 9-11] . 

The factor of urban-rural distribution was important because it was as- 
sumed that the differences in the facilities, functions, and problems of pra- 
ctice in urban versus rural areas could potentially result in a difference in the 
information needs and communications behavior of general practitioners in 
these areas. Wolf (1965) and Greenhill and Singh (1964) discussed the signi- 
ficant differences in the activities of /rural and urban practitioners. The dif- 
ference in types of problems and diseases commonly seen in urban and rural 
practices, the fact that the rural practitioner attempts more complicated 
general surgery and specialized surgery than the urban practitionor (Green- 
hill <S» Singh, 1965) would seem to indicate that there is a significant difference 
in the information needs of urban and rural practitioners. 

There also seems to be a marked difference in the number and kind of 
potential sources of information in urban and rural areas. The Task Force 
on Health Manpower (Pennell & Baker, 1965) reports that: 

For both economic and professional reasons, physicians tend to con- 
centrate in metropolitan areas. Such areas are usually characterized 
by high per capita income and population density and offer opportune 
ties for entree to large hospitals , frequent contact with hos pital staffs a 
and often access to medical teaching centers . 

(Underlining added for emphasis .) 

. . .In counties adjacent to these metropolitan counties the lower ratio 
of physicians may be in the larger medical centers , but this is not 
equally true for persons living in isolated counties. 

. • . The ratio of general practitioners to population is about the same 
in each of the county groups. For specialists in private practice, for 
physicians in hospital services, and for physicians in teaching, 
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research, and industry, the isolated counties are conspicuously low in 

comparison with the metropolitan areas [pp. 40, 46]. 

In summary, these classifications of different concentrations of popula- 
tion — greater metropolitan, lesser metropolitan, adjacent, isolated semi-rural, 
and isolated rural — could also serve as classifications or categories of the de- 
gree and ease of the availability of information to the general practitioner. As 
such, they would make natural strata for a sample in an information study. 

However, it was for all practical purposes impossible to stratify speci- 
fically on an urban-rural basis within each state. On the other hand, an 
analysis of the data available (Theodore & Sutter, 1966) indicated that a dis- 
proportionate stratified random sample would result in an adequate urban and 
rural representation. 

Urban areas in the United States include 80.4% of the general practitioners, 
while rural areas include 19.6%. The breakdown of urban and rural general 
practitioners in the eleven western states covered by this study is presented 
in Table 3. Urban areas include 85.8% of the general practitioners in these 
states; rural areas include only 14.2%. The wide variation between states 
is also shown in Table 3. In California 96% of the general practitioners are 
located in urban areas. The four states that would be considered primarily 
rural — Idaho, Montana, New Mexico, and Wyoming — together represent only 
9% of the total general practitioner population in the eleven western states . 

By decreasing the proportion of general practitioners in California and 
increasing the proportion in the ten other states, and by then taking a random 
sample from each state, it seemed reasonable to assume that the probability 
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of obtaining an adequate distribution of cases in the urban and rural categories 
would be increased. The logic of stratified sampling theory could then be 
applied in the analysis of the data (Seltiz, Jahoda, Deutsch, & Cook, 1959). 

After examining the percentage distribution of general practitioners in 
the eleven western states, it was decided to allocate California 50 per cent 
of the representatives it would have been allotted. Instead of 61 General 
Practitioners, 30 were included in the sample from California, and the remain- 
ing 31 General Practitioners were distributed among the other ten western 
states according to their proportion in the total general practitioner popula- 
tion. The final sample distribution of general practitioners shown in Table 4 
was derived following this procedure. 

TABLE 4 



Final Adjusted Sample Distribution 



State 


Nr. of ,G r Hs 


% of 
Total 


Proportionate 

Distribution 


Adjusted 

Sample 

Distribution 


Arizona 


40 r 


4% 


4 


7 


California 


5,928 


61% 


61 


30 


Colorado 


552 


6% 


6 


10 


Idaho 


239 


3% 


3 


6 


Montana 


236 


3% 


3 


6 


Nevada 


95 


1% 


1 


2 


New Mexico 


176 


2% 


2 


4 


Oregon 


591 


6% 


6 


10 


Utah 


248 


3% 


3 


6 


Washington 


1,001 


10% 


10 


17 


Wyoming 


138 


1% 


: i 


2 


Totals 


9,605 


100% 


100 


100 



Selecting the Sample 

After establishing the sample distribution among the different states, the 
actual drawing of the sample was undertaken. Each physician who met the 
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criteria discussed earlier was assigned a number. The required number of 
general practitioners for each state was selected from the population by 
pulling random numbers from two cans which corresponded to the row and 
column numbers of the table of random numbers. In this 7 /ay the designated 
sample of one hundred general practitioners for the eleven western states 
was chosen. 

Contacting the Sample 

Despite the opinion of many authors and the MIP Advisory Committee, 
it was concluded that the physicians' responses would be considerably high- 
er to the Medical Information Project program than to other types of post- 
graduate activities for the following reasons: (1) there would be a predisposi- 
tion to respond favorably if the assumptions were sound that time involved, 
convenience, travel, and cost are major factors which deter r\any physicians 
from postgraduate education; (2) the effective response rate of doctors to 
surveys and projects seems to be somewhat higher than that of other profes- 
sional groups. Parten (195(1) reported that one survey of M.D.'s in New 
York received approximately 50% returns without follow-up. Also acting as in- 
centives were the general novelty, interest, and practical possibilities of 
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this method of communication. 

It was necessary to develop an effective strategy of reaching the doctor 
and ensuring a high rate of participation. The importance of this problem is 
dealt with in the social sciences, (Slocum, Emprey, & Swanson, 1956; Linsky 
& Spendlove, 1967). The strategy which was developed employed techniques 
derived from the social sciences, advertising, and other practical and theore- 
tical communication procedures . 

The barriers which prevented effective communication with the physician 
were analyzed. The main problem was to attract his attention. The communica- 
tion demands on the doctor are horrendous. A letter can easily get buried in 
his daily mail. Medical Marketing (1960) reporting the sheer volume of mail 
stated that an average general practitioner in 1960 "received a total of 5,215 
mailings [p.,10]." A study conducted by Mark Dresden, Jr. (1960) found that 
a "doctor gets, on the average, seventy or more direct mail promotions a 
week [p . 4] . " 

In order to counteract this direct mail problem, a telegram was sent to 
each of the 100 physicians notifying him of his selection as part of the sample 
and informing him that a letter was to follow which would describe the project 
further. The letters, explaining the project objectives, activities, and time 
requirements and requesting the doctors' participation were mailed two days 
after the telegrams were sent. These letters were typed on official MIP paper, 
bearing the letterhead of the university, and personally signed by the director. 

A printed, self-addressed » stamped postcard was provided for their responses . 

A follow-up letter, again asking for their participation, was mailed to 






those doctors who had not returned the postcard within two weeks. A tele- 
phone call was then made to those who failed to respond to the second letter 
after two weeks. 

A letter of appreciation was sent to those who agreed to participate. For 
each general practitioner who declined, a name was drawn randomly from the 
same state to replace him. The same procedure for contacting the original 
sample was used for each substitute. 

Copies of the telegram, postcard, and the two letters used in contacting 
the sample are included in the Appendix. 

Analysis of Responses 

Five telegrams from the original sample of 100 general practitioners were 
not delivered, two because of death and three for unknown addresses. Immediate 
replacements were randomly selected and contacted. The responses of the 
general practitioners contained with the first letter were distributed as follows: 

First Letter 



Responses 


Nr. of GPs 


% of GPs 


Accepts 


66 


66% 


Declines 


10 


10% 


No Answer 


24 


24% 



The twenty-four doctors who had not yet responded by the end of the 
second week were the target group for the second letter. Of these fifteen 
agreed to participate in the study and nine declined to do so. 

The total figures for the original sample of 100 doctors were eighty-one 
(81%) who agreed to participate and ninetten (19%) who declined participation. 
In order to fill the nineteen slots, it was necessary to contact twenty- seven 
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doctors, of whom nineteen accepted and eight declined participation. 

Four of the physicians who declined were not eligible, by reason of 
retirement or specialization, to be part of the sample. Thus, 123 physicians 
were contacted in order to arrive at a sample of 100 general practitioners, 
giving an overall acceptance rate of 81.3%. 

Table 5 shows a summary of the data on the results of contacting the 
sample physicians and their responses. No elaborate analysis of the simi- 
larities and differences between first and second acceptors was conducted. 
Problem of Decliners 

A question of considerable interest arises as to information on those who 
did not respond: Did the twenty- three (23) doctors who declined to partici- 
pate as part of the experimental group in the study constitute a possible 
self-selection bias in the final sample ? There is a field of thought that 
purports *hat those physicians who agree to participate in such programs as 
this are the same ones (the literature indicates the figure is between 20% and 
30%) who account for most of the physician- hours spent in postgraduate pro- 
grams; whereas , those who refuse may very well be among those in most need 
of some form of postgraduate work. Vollan (1955) found that: 

Although the majority of physicians do take some postgraduate 
work from time to time , with a few receiving abundant training 
of this kind, there still remains a sizeable group — probably between 
30 and 50% — of practicing physicians who never take any [p. 47]. 

In order to determine whether those who declined to participate were 

different from those who agreed to participate , the two groups were compared 

on the demographic data (age, year of graduation from medical school, etc.) 
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SUMMARY OF DATA ON SAMPLING 



Original Sample 100 

Substitutions for nondelivery of telegrams: 

Due to death 2 

Due to moving 3 

# of telegrams sent originally. . 105 

FIRST LETTERS : First contact — number sent 100 

Total accepts 66 

Total declines 10 

No answer 24 

SECOND LETTERS : Second contact — number sent 24 

Total accepts " 15 

Total declines ( 9 

Of Original Sample; 

ACCEPTS : Totals from letters 1 & 2 81 

DECLINES : Totals from letters 1 & 2 19 

Total Substitutions for Declines (27) 

FIRST LETTERS : Number sent 27 

Total accepts 16 

Total declines 5 

SECOND LETTERS: Number sent 6 

Total acc epts 3 

Total declines 3 

Of Substitutions for Original (19) Decliners; 

ACCEPTS : Totals from letters 1 & 2 19 

DECLINES: Totals from letters 1 & 2 8 




44 



obtained from the AMA Directory. This comparison did not yield any signifi- 
cant differences between the two groups . 

In addition to this comparison, the physicians were asked to indicate 
their reasons for declining. Of the doctors who were eligible to participate 
and declined, ten volunteered their reasons by jotting them on the response 
postcard. Three gave personal reasons such as illness and vacation; four 
stated that they were "too busy." It is interesting to note that three gen- 
eral practitioners mentioned extensive postgraduate activity as a reason for 
not participating in this study. The list of reasons for declining is shown 
in Table 6. 



TABLE 6 



• Reasons for Declining 


Reasons for Declining 


Number 


Serious illness 


2 


Too busy 


4 


Extended vacation trip 


1 


Too much postgraduate education 


3 




10 


No reasons given 


13 


Total 


23 



On the basis of information available, it was concluded that the physi- 
cians who declined participation in the Medical Information Project were not 
significantly different from those who accepted. 

The Control Group 

The control group was drawn and selected on the ne basis as the 
experimental group of physicians. This group was activated, towever, at the 
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time that the pre-program questionnaires were mailed. These physicians ini- 
tially were contacted and their cooperation solicited by sending them a night- 
letter informing them of their selection as a member of the control group for 
the study. This was followed by sending them the pre-program questionnaire 
with a covering letter explaining the objectives of the project and requesting 
their help in serving as a medtiber of the control group by completing the question- 
naire. One follow-up letter was sent to non-responding physicians, and if the 
physician still did not respond a new name was drawn to replace him. 

It was expected from the beginning that the effective response rate from 
these physicians would be somewhat iower than for the experimental group, 
since many doctors are inundated with requests to complete questionnaires from 
the pharmaceutical industry, professional organizations, etc. Even so, the 
response rate for the control group was 54%. This rate is still higher than the 
expected response rate to mailed questionnaires indicated by Riley (1963, p. 190) , 
Seltiz et al (1962, p. 241) and Simon (1962, p. 249) where the range was be- 
tween 10 to 50 percent. 

Summary 

The procedures of contacting the experimental and the control groups have 
been described. Several conclusions can be drawn in this regard. A project 
that has something to offer a professional, even if it involves certain commit- 
ments .in terms of time requirements has a good chance of obtaining a high rate 
of participation. Furthermore, as was pointed out by Slocum, Emprey and 
Swanson (1956), the method by which the sample is approached seems to increase 
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the rate of positive response ; (a) establishment of professional and social 
utility of the survey, (b) detailed descriptions of the project, (c) several 
follow-ups of non-respondents, (d) university sponsorship, (e) personalized 
contact. Finally, physicians seem to be a mere receptive population then 
the general population, a conclusion which was previously drawn some 
twenty years ago by Parten (1950). 
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PHASE II: PRODUCTION AND PROGRAM DELIVERY 



Distribution of Machines and Training Materials 

In May, 1967, work was begun on the production and packaging of the 
introductory training program for the participating physicians. This program 
covered the general concept of the Project and the operation of the Hoffman 
audiovisual machine. It also provided practice to familiarize the physicians 
with the concept of programmed instruction. This program was delivered with 
the Hoffman machines . 

The Hoffman equipment was installed in the doctors' offices during 
the months of November and December, 1967. Originally, it was planned 
that a Hoffman dealer would install the machine in the physician's office, 
check for any problems with the equipment, and demonstrate its operation 
to the physician. However, it was discovered that the machines had been 
shipped to the participating physicians and that the dealers had visited 
only about twenty of the doctors, and this only after the individual physician 
had called the dealer n his area. Consequently, the doctors were contacted 
to find out if they needed help in learning to operate the equipment and if 
the equipment was in good working order. Several of the machines didn't 
work properly, and were repaired or shipped back to Hoffman and replaced 
with new~o?r&s^^However , at least one participant quit the Project during 
this period because of problei^'withJhe^ equipment and servicing by Hoffman. 



With the experimental group established, the general design determined, 
the production processes operative, the machines installed, and the pre- 
program questionnaires administered to both the experimental and control 



group of physicians, the Production or Program Delivery Phase became 
operable in April, 1968. 

Design, Production and Distribution of MIP Programs 

The preparation of fifteen audiovisual programs designed for viewing 
by general practitioners was an exacting and difficult task. It involved 
the transformation of complex medical and scientific knowledge into a 
new form: a concise and vigorous presentation in a new medium. Fifteen 
audiovisual programs were designed, produced and distributed to the 
participating physicians during the period from April, 1968, through 
August, 1969. Figure 8 lists the programs produced by the Medical Informa- 
tion Project with titles, dates, and personnel involved in each production. 
Basic Responsibilities of the Production Team 

The production of the programs required the formation of a smoothly 
functioning team which could convert important medical information into an 
audiovisual format. As noted earlier, the production team consisted of jthree 
people: a producer/writer , an artist, and a photographer. The medical 
consultant was a key member of the team who provided the basic outline 
of information to the program producer, researcher, artist and photographer, 
thus placing him in the difficult position of having to impart his specialty 
to the production group in a short period of time. 

The basic responsibilities of the production component were as 
follows: 

1. Design of Programs : statement of behavioral objectives , content 
research, selection and preparation of artwork and photography, production 
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of scripts and storyboards, production of preliminary programs. 

2. Coordination of Input from Consultants and Supplementary 
Personnel : scheduling meetings with consultants, coordinating researchers' 
efforts, obtaining guidelines and criticism from physician consultants and 
Medical Advisory Committee, validating and revising programs through local 
physicians, working with hospital and other sources for visual material. 

3. Coordination of Production of Final Program : integrating all feed- 

's. 

back elements for final revisions, completing final photography and art work, 
recording sound track with professional narrator. 

4. Coordination of Reproduction through Commercial Sources : distri- 
bution of final programs to various commercial companies for pressing and 
labeling of records, reproduction of filmstrips, encapsulation of filmstrips, 
punching filmstrips, pulsing records, printing of program booklets, etc. 

5. Coordination of Program Distribution : checking quality of 
finished products, packaging materials, arranging delivery through com- 
mercial mailing company. 

6. Revision of Processes and Products : interpreting data from 
program booklets (program evaluation forms, tests), analyzing design and 
production effectiveness, reorganizing basic strategies and content 
approaches as needed. 

Programming Strategy 

The main points of the MIP programming approach which determined the 
tactics selected in the production process and the resultant modifications 
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of these tactics were: 
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1. The programs were presented in audiovisual form using a machine 
that could sit on a desk in a doctor* s home or office. 

2. The medium was projected, stil], colored pictures, reinforced by 
sound. 

3. The programs were approximately twenty-four minutes in length, 
divided into units of five to six minutes each. A program usually 
had self-contained parts. 

4. The program presented content in sequences of frames which 
were followed by a frame on which the machine stopped auto- 
matically and which directed the physician to a program question 
in the booklet. The booklet required some sort of response — 
labeling a diagram, filling in a blank, etc. The physician 

then restarted the machine for the next sequence. At the 
completion of the program, a summary frame was presented. 

The physician then took a short content test and evaluated the 
program . 

5. These programs had the following type frames: 

a. Motivation frames (MF)--attempted to motivate the doctors 
with reference to the content and objectives. 

b. Objective frames (OF) — related to the major objective of the 
programs; namely, to induce the doctor to change his com- 
munication behavior by seeking more information on a sys- 
tematic basis. 

c. Content frames (CF) --contained information. 

d. Summary frames (SF) — summarized the main point of a 
presentation sequence. 

e. Response frames (RF) — referred the doctor to the program 
booklet where he was asked to respond to a question or 
problem. 

f. Answer frames (AF)--immediately followed the response 
frames and provided the correct answer to the question or 
problem. 

6. Essentially, the program questions were used for three purposes 
in the Medical Information Project programs: 

a. To review information. This type asked the doctor to review 
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information previously covered in the program. 



b. To preview information. In this case, a general question 
was asked that referred to information not yet covered in 

the program. The answer was given in the frame immediately 
following, and then the topic was developed. 

c. To branch ahead . A question was asked on the material 
to be covered in the remaining frames on that particular 
filmstrip. If the doctor answered correctly, he was given 
the option of pushing the eject button, thereby by-passing 
the remaining frames on that filmstrip, or he could review 
the rest of the filmstrip. These questions were more specific 
than those of types a or b. 

The Production Process 



In March, 1967, the MIP Research Memorandum Number 3: Strategy 
and Tactics for Program Preparation was written as a guide for the production 
team in designing the instructional units. It analyzed and attempted to 
define the relationship between the production team and the medical 
consultant in order to arrive at the best possible content programs. 

The tactics outlined were introduced with a precaution: "These 

*N 

tactics must be viewed, however, as tentative and subject to revision as 
experience progresses. " During the course of the production phase and the 
preparation of fifteen programs, the production team acquired this experience 
which necessitated revisions in the concept of the tactics involved. The 
tactical changes that evolved in practice centered around the roie of the 
medical consultant in the specific stages involved in program preparation. 

In attempting to describe the production process, it is difficult to list 
specific steps in a specific, linear order. For example, no two consultants 
had the same style of working. One may have worked many hours on the 
actual scriptwriting, whereas another may have been primarily concerned 
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with approving material that was assembled by the production crew. Although 
the steps are presented in linear form, the process was really much more 
fluid than described. 

There was a give-and-take between members of the team as each 
played a role in trying t.o translate the specific content to the medium. 

The production group helped the consultant think in terms of the medium 
rather than as if he were writing an article or giving a lecture. All worked 
together in an attempt to refine and clarify the content into a concise teach- 
ing unit. The production process (shown in Figure 9) covering the stages 
from content selection through delivering the finished product to the partici- 
pating general practitioner, is described in detail in the following pages. 

Content Development . The general content areas from which the 
specific program topics were selected were established at the first meeting 
of the Advisory Committee on March 22, 1967. These areas were chosen on 
the basis of what the panel members felt was information needed by the 
average general practitioner to keep abreast of the changes in medical 
knowledge and which would have immediate practical implications for the 
physician's medical practice. Twenty content areas were originally listed, 
more than would be produced, in order to make substitutions if problems in 
obtaining consultants or technical barriers arose. Six specific topics were 
chosen initially from these twenty content areas. Specific topics for 
remaining programs were decided on the basis of feedback from the partici- 
pating physicians through interviews, the questionnaires, and the program 
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evaluation forms. 
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Once the decision for a specific program topic had been made, a 
consultant was recommended by Dr. Phil Manning or other members of the 
Advisory Committee. The consultant was contacted by Dr. Manning to 
secure his participation and cooperation. The consultants responded 
favorably in almost every case. A confirming letter was sent to the con- 
sultant explaining the proposed topic, the pay, and an estimate of the time 
which would be involved. The following items were also enclosed,' 

1. The MIP information leaflet, stating the concept 
and objectives of the project. 

2 . A copy of Strategy and Tactics for Program Preparation 
(this was eliminated from the packet during the last 
seven programs because of changes in methodology). 

3. A copy of Mager's Preparing Instructional Objectives . 

An appointment was made for the project producer and the researcher 
to meet with the consultant to establish the concepts that would be covered 
in the program. During the initial meeting with the consultant, the producer 

and researcher attempted to clarify format and the length of time that an average 

program should run. The consultant in most instances attempted too broad 
a coverage to be presented in a single program. 

An important outcome of this initial meeting was a statement of 
behavioral objectives for the particular instructional unit. The behavioral 
objectives were defined in terms of what the physician should be able to do 
after viewing the program; what points of knowledge he should take away 
from the program. 

The consultant recommended sources of informati on from which the 
topic could be researched and a content outline written for the roucjh script 



and storyboard. The researcher/writer had access to the USC School of 
Medicine Library which has an extensive collection of medical volumes. 
The researcher, working with the producer, collected as much factual 
information pertaining to the topic as possible. The collection of this 
material often took as long as thirty days. The writer also obtained a wide 
range of materials to review such as films, reference books, pertinent 
articles, pamphlets, drug company brochures , slides, transparencies, and 
illustrations. Materials were examined and key items were submitted to 
the producer. Although originally it was thought that the content outline 
would be written in close conjunction with the consultant, the consultants 
were usually too busy to do this. The problems involved in scheduling 
meetings convenient to both the production staff and the consultants were 
often exceedingly difficult. The writer finished a detailed outline, 
carefully noting the relationship to the initial objectives and time distribu- 
tion in the program. Key points were also noted for possible programming 
frames. 

The producer presented a preliminary program storyboard to the 
medical consultant who would modify, recommend, delete, and generally 

focus the content of the program to his perception of the general practi- 
tioner's needs. 

The following summarizes the preliminary activities of the production 
team in the content development for a particular topic; 

a. . Selecting specific topic with the consultant from the 

general content area. 

b. Clarifying and defining specific behavioral objectives. 



c. Researching the topic and compiling notes based on 
the consultant's recommendations. 

d. Developing general content outline. 

e. Suggesting possible treatments . 

f. Discussing with the consultant the objectives, the 
content outline, and exploring possible treatments which 
would lead to a draft script and formal storyboard. 

Development of Storyboard . In analyzing the subject content for the 
preparation of the storyboard, the production team worked from a set of 
notes, an outline, an essay, a complete script, or any variation of these. 

It was in the development of the storyboard that the basic approach to the 
presentation of the material, audiovisually , was determined. By working 
on paper, with the artist sketching appropriate visuals, a variety of 
approaches both to overall design and individual frame design were con- 
sidered. The producer wrote a preliminary program based upon the research 
material and then wrote the first draft of the storyboard narration, defining 
the relationship between the audio and visual channels of communication in 
each frame. 

Working closely with the artist and photographer, the producer 
tightly edited the commentary making continual adjustments in response to 
creative suggestions by the production team and the consultant. During this 
time, the consultant, working with the producer, developed the content 
test which was to accompany the program. The producer also developed a 
draft of the program questions for reinforcement of concepts that were to be 
included in the booklet. 

At the end of this stage, the following had been completed: 




a. The producer completed the storyboard with each frame 
consisting of the artist’s rough sketch and typed com- 
mentary. 

b. The medical consultant recommended specific changes, 
and gave preliminary approval. 

c. The producer developed response and answer frames 
in consultation with programing advisors. 

d. The consultant, with the producer, developed the 
content test. 

e. The producer drafted the program booklet, including the 
program questions and content tests. 

Validation and Pre-Production Testing . Program validation is considered 
the key item in the technology of programing instructional materials. Audio- 
visual program validation is somewhat more difficult than the validation of 
printed verbal materials. The original plan for both program validation and 
pre-production testing called for the use of storyboard pencil sketches of 
the artwork and photographing these for slides for the visual frames, a 
technique which has proved effective in determining the instructional 
strengths and weaknesses of a film (Rose and Van Horn, 1956). To these 
would be added live photographs, particularly where precise scientific 
photography was required. 

Pencil sketches were used in the early programs, and later colored 
felt tip pens were tried. However, the method which ultimately proved 
to be most effective, as well as feasible in terms of time and expense, was 
the use of white chalk on chalkboard, photographed on black and white 
film, with the negatives mounted in 2" x 2" slides. The resulting image 
was a dark line on a gray background. With this process the visuals could 



be shot, developed, and mounted in slides on the premises within an hour, 
if necessary. 

In-house narration in a form similar to the final product was taped to 
accompany the preliminary slides. Question frames, answer frames, and 
other configurations that were to appear within the program were included. 
Although many producers of educational and medical films test their material 
in the script and/or storyboard stage, it was felt that most laymen in the 
field of motion picture and slide production usually cannot visualize the 
final treatment of material solely from scripts or storyboards. Therefore, 
the storyboard was taken one step further. It was presented visually by 
slides of the rough sketches of artwork and photographs and accompanied 
by the approximate sound track that would be used in the final program. 

This procedure worked well as a cost-effective compromise between 
i nexpensive paper scripts and a costly full-blown production of a prelimi- 
nary program that might very well require revision. 

The program was shown in this form to the consultant, the Advisory 
Committee, and then validated and pre-tested with a panel of three to 
five physicians in the Los Angeles area. The visuals and the narration 
were tested in combination this way since any change in one channel could 
alter the effectiveness of the other, and the "feedback" must relate to the 
whole program. This procedure also helped to avoid the common pitfall 
of encouraging attention to the verbal content rather than the visuals, 
following Ruhe's (1953) suggestions for overcoming the prejudice of medical 
consultants for emphasizing verbal type lecturing over visual content. After 
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the program was shown to the physicians, their comments and criticisms were 
recorded. The evaluation forms were studied and, if necessary, the audio, 
the visuals, or both channels were revised and the programs re-tested. 

One example is described to demonstrate the importance of pre- 
production testing. In pre-testing "Assessment of Maturity in the Newborn", 
the most frequently missed question was in relation to the feeding of a low- 
weight newborn. A change in the recommended number of calories per kilogram 
has taken place in the last few years. While the correct number of calories 
was mentioned in the program, it was apparently not stressed enough to 
change previous knowledge. After revision, with emphasis added to this 
point, the physicians apparently learned the intended information. It was 
concluded that areas of recent changes in medical knowledge needed 
special emphasis rather than simply presenting the new information in a 
straight-forward manner. 

The following summarizes the activities during the validation and 
pre-production testi ng stage: 

a. A set of 35mm slides was prepared consisting of photo- 
graphs of the rough chalk sketches, and the actual photo- 
graphs to be used, when available. 

b. The narration was recorded on tape, in-house, and the 
audio track was pulsed to approximate the timing suggested. 

c. The consultant approved, or recommended, specific changes 
on the basis of the slide-tape presentation. 

d. The slide-tape program, with the program frames and content 
tests, were tested on a panel of three to five physicians. 

e. The producer made revisions in the program, response frames, 
and content tests, if necessary, as suggested by the con- 
sultants and program advisors. 
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f. The consultant gave final approval of the program. 

g. The Advisory Committee gave preliminary approval of the 
program . 

Production of Final Program . After validation and revision , production 
of the final visuals was undertaken. Visuals were secured from a wide 
variety of sources. These included both original clinical and surgical 
sequences, full color art work, and half-tone reproductions. Some of the 
program topics required photographs almost exclusively; others necessitated 
a great deal of art work. For example, several topics, such as "Skin 
Tumors," "T. Rubrum," and "Jaundice in the Newborn" lent themselves 
more readily to actual color photographs of the conditions. On the other 
hand, such topics as "Family Planning" and "Tranquilizers" were composed 
primarily of art work, such as graphs, charts, etc. 

After the final set of 35mm slides was prepared, using the in-house 
taped narration , the program was shown to tho Advioory Committee for 
formal approval. Any final changes were made as recommended by the 
consultant and/or the Advisory Committee, and the approved set of slides 
were then copied with the repronar on negative film on the standard half- 
frame camera. This film was sent to RGB Labs (Identicolor) for processing 
into a master filmstrip. 

The physical production of the final program is summarized as follows; 

a. Final art work was produced and photographed, final 
photography was completed, and existing slides were 
copied. 

b. Final production approval was obtained from the 
Advisory Committee. 
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c. The approved set of 35mm doulbe frame, 2" x 2" slide 
transparencies were copied on the repronar. 

d. An internegative was processed to provide the master for the 
filmstrip reproduction. 

Commercial Reproduction . After the color intermediate of the filmstrip 
was approved by the MIP production team, the intemegative was sent to 
Deluxe General where it was looped and reproduced on a thousand-foot strip 
of positive release print, a quantity sufficient to supply the physicians 
participating in the project. The release print was sent to Hoffman where 
the final filmstrips were cut at the appropriate intervals and encapsulated 
in the plastic holders used in the Hoffman device. These holders were 
then punched to activate the prgramaiing function at the appropriate frames. 

While the filmstrips were being reproduced, a professional narrator, 
George Walsh, recorded the script at RCA sound studios. This magnetic 
tape was edited at RCA, a continuous 50-cycle tone was added, and the 
tape was pulsed in accordance with the script. Breaks in the tone activated 
the mechanism which either changed the frame, or in the presence of a hole 
punched in the plastic filmstrip holder, stopped the mechanism in the 
programming mode. The 7 " records were then pressed and the labels 
giving the program number and the title for the records were printed. 

At the same time that the records and filmstrips were being reproduced, 
the content test, program questions, title sheets, credits, and any graphs, 
charts, etc. were typed and edited. The covers for tho program booklets 
and the program evaluation forms had previously been printed and remained 
the same throughout the course of the project. The new pages were sent 
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to Don Figge Associates to be printed and collated. 

The following summarizes the activities involved in the commercial 
reproduction of the MIP programs: 

a. The color intermediate of the filmstrips was approved; 
the internegative was looped and printed; the filmstrips 
were cut, encapsulated and punched. 

b. Narration was professionally recorded and edited; pulse 
was added to the tape; records were pressed and labels 
printed. 

c. Content tests, program questions, and other materials 
were typed and edited; program booklets were printed and 
collated. 

Packaging and Distribution . When all of the components for a 
given program were reproduced and returned to the Medical Information 
Project production office, a quality control check was undertaken. 

Records were selected at random from the run and tested on the Hoffman 
machine in conjunction with a random selection of filmstrips. Several 
factors were checked, including correct pulsing for synchronization of 
record and filmstrips, and proper labeling of records. Some serious errors 
were caught in this way. In one instance, RCA had pressed both sides of 
one record with the same narration. The program booklets were also 
checked for errors in printing or collating. Each of the booklets was number 
coded . 

After this check, each of the filmstrips was labeled with its appropriate 
number (1-4); included on this label was an arrow which made it easier 
for the physician to know which end of the filmstrips to insert into the 
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projector. 



Then began the task of assembling the packets. The albums usually 
contained the following items: 

a. Four filmstrips, labeled in sequence. 

b. Two records, both sides appropriately labeled. 

c. Program booklets, containing program questions, evaluation 
forms, and content tests. 

d. Postcard requesting further information. 

e. Self-addressed, stamped return envelope for return of forms . 
Once the packages had been assembled, they were turned over to Minuteman 
Mailers along with album mailers and address labels for shipping to the 
participating physicians. 

Problems Encountered in Production 

The production process was originally planned to encompass an 80-day 
cycle. However, this varied significantly with problems that were en- 
countered in the production of each of the specific programs. In any system 
the operation does not always run as smoothly as it was conceived on 
paper. The design of a system attempts to take into account as many 
factors as possible. However, many problems arose that could not be 
foie scon. 

Although the original MIP schedule indicated that a program would be 
sent every two weeks to the participating physicians, many factors inter- 
vened to alter this plan. Most of these delays involved situations largely 
out of the control of the MIP staff. The best that could be done was to 
attempt to solve them when they arose as quickly as possible to minimize 



the delay. 
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Personnel turnover can cause an undetermined amount of slippage. 

Three producers were Involved in production at different times during the 
course of the project. Some delay occurred during the period when it was 
necessary to orient the new producer with the tasks in progress, processes, 
involved, etc. 

Many delays were encountered due to the extremely busy personal 
schedules and illnesses of consultants, narrator, Advisory Committee 
members, and validating physicians. Each time a delay in a single area 
was encountered, the production of that program was thrown off schedule, 
and further compounded the problems in other individual schedules and 
appointments. Scheduling meetings of the Advisory Committee often caused 
delays up to two weeks in which production necessarily came to a halt 
awaiting final approval for prepared programs . 

Much delay was caused by the difficulty in arranging meetings with 
the consultants. The consultants who agreed to work on this project were 
paid a very modest honorarium. The amount of time involved for them 
varied from 20 to 40 hours. These consultants woro hlahly respected men 
and women in their fields of medicine. Because of the pressing icquinsmcuLs 
of their other professional duties, appointments with the production team 
were often not kept or cancelled, resulting in a delay of weeks ( or even 
months in a few cases) between initial contact and eventual production of 
the program. 

For this reason the production team had at least three programs in 
various stages of production at any give time to minimize consequent delays 
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in production. Working on a number of programs simultaneously had its 
disadvantages, however. The production team was required to fragment its 
attention and efforts over several programs at various stages in production 
to utilize time as efficiently as possible under the circumstances, rather 
than being able to concentrate totally on one program to assure the smooth 
and orderly progression of the nyriad of production details required. In spite 
of this, the programs remained high quality productions. 

The distances involved should also be mentioned as this caused a great 
deal of travel time by the production team. The medical campus was located 
10 miles from the main campus where the MIP production office was located, 
causing much travel for every meeting with consultants. Travel time was 
involved in going to each of the physician's offices who were involved in 
validation of the programs. Furthermore, a round trip of nearly 70 miles was 
required to take each release print to Hoffman Electronics for encapsulation, 
involving costly time of one of the production team members. 

By far the most difficult problems encountered were in the commercial 
reproduction of the programs, where the production staff was literally at 
the mercy of the companies involved. Delays were encountered in the com- 
mercial pressing of the records and the encapsulation of release prints 
because each company had larger orders to fill. For example, the number 
of records required for MIP programs was a small run (100), and RCA often 
took from three to four weeks from the time of recording to finish the records. 
The small run of MIP filmstrips would be worked into the schedule of Hoffman 
Information Systems as per their convenience. Such delays caused mailing 



dates and other basic time schedules to be jeopardized, thus holding up the 
entire production process. 

The following illustrate some of the incidents which caused unavoidable 
delays over which the production team had no control: 

1. Printer injured hand causing one week delay in printing 
program booklets. 

2. Company punched filmstrip holders incorrectly and broke 
punch causing one week delay. 

3. Loss of film and negative by processor causing three 
week delay. 

4. Three weeks delay caused by record plant being closed 
down for repairs . 

5. Narrator cancelled recording date because of laryngitis 
causing one week delay. 

6. Record cutter ill; record labels delayed in printing; wrong 
labels on records resulting in total of two week delay. 

7. RCA pressed both sides of record with same narration resulting 
in delay of three weeks. 

8. Illness of consultant resulted in six week delay. 

9. Senior consultant turned project over to junior consultant and 
change in emphasis resulted in three week delay. 

10. After six-month production period, program was vetoed by the 
Advisory Committee, and consultant was replaced. 

R ecommendations on the Production Process 



After completing the production of fiftoon inou-uotiwnai piogmms. a 
few recommendations can be made with respect to the production process. 
First, the production unit would function with more efficiency if they were 
located nearer to the consultants, in this case, the medical school campus. 
This would encourage closer cooperation between the consultant and the 



70 



product ' on team and would alleviate some of the problems of travel and 
scheduling meetings. 

Second, adequate lead time should be allocated so that it would be 
possible to concentrate efforts of the production team on one program from 
start, to finish, rather than overlapping work on several programs at the same 
time. Another possibility would be to have two production teams working on 
separate productions. This, of course, would have to be decided in terms of 
economic feasibility. 

Third, every attempt should be made to validate and pre-test programs 
prior to the final production. MIP' s experience with using slides of a 
photographed black and white storyboard with taped narration proved more 
valuable in pre-testing productions of this nature than relying solely on 
reading the script. The validation aind pre-testing of productions should 
be done with physicians who will respond frankly about the programs. The 
physicians who assisted the Medical Information Project in pre-production 
testing seemed reticent tr> he as ciitioal an tlicry might have been. 

Finally, the program researchers were generally graduate students 
from the Department of Instructional Technology, and except in rare instances 
had little or no background in medicine. These students were well-versed 
in the steps required in program production. However, at the level at which 
they were working, programming ability Vk&s not the important or perhaps 
even the necessary area of expertise. Therefore, it was concluded that 
medical students would be in a better position to grasp the relative impor- 
tance of different aspects of a medical topic for inc’usion in the content 
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outline. In addition, they would be more familiar with the concepts and 
terminology involved in researching a content area. 
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Figure 10 

Medical Information Project Production Building 





Figure 11 

Delux General Film Laboi&tories - Filmstrip Printing 
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? Figure 12 

j Medical Information Project Narrator George Walsh 




Figure 13 

RCA Recording Studio* Hollywood, California 
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Figure 14 

Shooting Validation Frames 

First Visual Production of MIP Programs 
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Figure 16 Sorting Slides for Storyboard 





Figure 17 

Working on Storyboard with Consultant 
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Medical Information Artist, Peggy Laird 




$9 



77 




Figure 19 

Shooting Final Art for Medical Information Project Programs 
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Figure 22 

MIP Program Package — Album Containing Completed Materials As Sent To Participating Physicians 
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Phase III 
Evalua tion 

Findings and Recommendations 

Introduction 

Three broad categories of interest were analyzed in this study in addition 
to the developmental study which resulted in the choice of the devices and the 
production of programs and in their improvement before project initiation. 

Next are presented the kinds of hypotheses relevant to these three areas, the 
data collected and analyzed to test these hypotheses, and the assessment of 
the relative success of each of the three areas of interest. In addition, there 
will be some comment on other problems which were not formally measured 
in the project. 

Content Learning 

Of primary interest is the question of whether the subjects actually 
learned a significant part of the content presented to them in the program. 

The scheme of testing has been described earlier and Table 7 summarizes 
the results. The practical problems of attrition, passage of time, and equip- 
ment problems still left an experimental group of 73 and a control group of 
36 with results on 9 programs which could be evaluated. 

The hypothesis tested is the following. Programs which are effective 
in the nine content areas chosen will be such as to result in significantly 
higher achievement on the post-test by the experimental than the control 
group. The t-test chosen to do the analysis reveals achievement differences which 




are statistically significant beyond the 1% level. Further, the consideration 
of the 103 separate items of the 9 content tests may be pooled into one 
"Grand Content Test" and retested. Here again, the t-test value of 17.42 
is signifcant beyond the 1% level. 

The manner in which these data were collected is described in an 
earlier chapter; the precautions which were taken to ensure validity of the 
content test seem sufficient to consider the hypothesis verified. 

The sample described on Page ;jgwas drawn as indicated there. Table 
8 indicates the resulting experimental and control groups, with the greatest 
attrition occurring at the point of returning the content tests and post- 
program questionnaires. Tables 9 through 15 illustrate various comparisons 
among the 97 experimental and 85 control people with regard to age, years of 
professional experience, sex, distances from medical schools and hospitals, 
hours worked per day and type of practice arrangement. 

Tables 16 and 17 compare the sample with the population it reflects, 
using the original 100 experimental and 85 control physicians. The valid 
comparisons which could be made are discussed below; however, the generali- 
zation is that there is no reason to suppose differences which render the study 
less fruitful. 

Of course, one must look at the final sample of 73 experimental and 36 
controls, since the t-tests are based on that group rather than the original 
sample. These demographic datav/sre gathered on th£ subjects as Indicated 
in Table 18 through 22, to see if there were other biaees present in the final 



sample. 



TABLE 7 



RESULTS OF CONTENT TESTS, EXPERIMENTAL AND CONTROL 
GROUPS, FOR NINE CONTENT AREAS 



No. 

of 

Items 


Title and No. 
of Program 


Experimental 
(N = 73) 

X SD 


Control 
(N = 36) 
X SD 


t value 


14 


1 . 


Cardiopulmonary 

Resuscitation 


11.33 


1.01 


9.44 


2.20 


t=6. 07** 


9 


2. 


Vaginitis 


8.32 


0.72 


6.61 


0.98 


t=10. 18** 


15 


3. 


T. Rubrum 


12.73 


2. 15 


4.83 


1.99 


t=22 . 92** 


16 


4. 


Family Planning 


13.62 


1.96 


8.56 


2.48 


t=ll. 47** 


4 


5. 


Routine Gyn Exam 


3.40 


0.68 


2.25 


0. 76 


t=7. 91** 


10 


6. 


Assessment of 
Newborn 


8.66 


1.76 


3.28 


2.09 


t=13. 95** 


6 


7. 


Skin Tumors 


5.64 


0.53 


2.81 


1.20 


t=16. 95** 


19 


8. 


OB Emergencies 


16.01 


2.32 


8.08 


2.99 


t=15. 86** 


10 


9. 


Newborn Taundice 


8.86 


1.46 


6.50 


2.09 


t=8. 20** 



TOTAL; 



103 



ALL TESTS 



88.57 12.59 



52.36 16.78 



t=17, 42** 





EXPERIMENTAL 


CONTROL 


INITIAL CONTACT 


123 


147 


Affirmative to participation 


100 


85 


Received pre-program questionnaire 


100 


85 


Returned pre-questionnaire 


97 


85 


Received post-program questionnaire 


93 


80 


Returned post-questionnaire 


77 


47 


Sent content tests (1-9) 


97 


80 


Returned content tests 


73 


36 



97 



84 



TABLE 9 

COMPARISON OF EXPERIMENTAL AND CONTROL GROUPS: GRADUATION DATE 



Dates 


Experimental (N=97) 


Control (N=85) 




Nr. % 


Nr. 


% 


1960-63 


4 4.1% 


7 


8.2% 


1950-59 


39 40.2 


30 


35.3 


1940-49 


37 38.1 


23(24) 


27.1 


1930-39 


17 17.5 


23 


27.1 


1920-29 


0 


1 


1.2 




Range = 1929-1962 


Range = 


1930-1963 




Mean = 1946 


Mean = 


1948 




Median = 1946.3 


Median 


= 1949.2 



TABLE 10 



COMPARISON OF EXPERIMENTAL AND CONTROL GROUPS: AGE 



Age 


Experimental 


Control 


Nr. 


% 


Nr. 


% 


60 - 65 


2 


2.0% 


11 


12.9% 


50 - 59 


35 


35.0 


31 


36.5 


40 - 49 


46 


46.0 


29 


34.1 


30 - 39 


17 


17.0 


14 


16.5 




Range = 


32-64 


Range = 


33-64 




Mean = 


49.00 


Mean = 


46.93 




Median 


- 49.50 


Median 


= 46.50 
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TABLE XI 



COMPARISON OF EXPERIMENTAL & CONTROL GROUPS: SEX 


Sex 


Experimental 
Nr. % 


Control 

Nr. 


% 


Male 


98 


98.0% 


82 


96.5% 


Female 


2 


2.0% 


3 


3 . 5% 



TABLE 12 



COMPARISON OF EXPERIMENTAL & CONTROL GROUPS: MILES FROM MEDICAL 



SCHOOL 


Miles 


Experimental 
Nr. % 


Nr. 


Control 

% 


0-9 


18 


18.6% 


14 


16.5% 


10 - 24 


11 


13.4 


14 


16.5 


25 - 49 


12 


12.3 


6 


7.0 


50 + 


54 


55.7 


51 


60.0 



TABLE 13 



COMPARISON OF EXPERIMENTAL AND CONTROL GROUPS: MILES FROM HOSPITAL 



Miles 


Experimental 


Control 






Nr. 


% 


Nr. 


% 


0-5 


87 


89.7% 


73 


85.9 


6-15 


8 


8.2 


7 


8.2 


16 - 25 


1 


1.0 


4 


4.7 


26+ 


1 


1.0 


1 


1.2 



9 ? 
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TABLE 14 



COMPARISON OF EXPERIMENTAL AND CONTROL GROUPS: 


HOURS WORKED 
PER DAY 


Hrs . per day 


Experimental 
Nr. % 


Control 

Nr. 


% 


0-5 


2 


2 . 1% 


1 


1 . 2% 


6-8 


19 


19.6 


15 


17.6 


9-11 


51 


52.6 


50 


58.8 


12 - 14 


21 


21.6 


16 


18.8 


15+ 


4 


4.1 


3 


3.5 



TABLE 15 



COMPARISON OF EXPERIMENTAL AND CONTROL GROUPS: TYPE PRACTICE 
ARRANGEMENT 



Type of Practice 
arrangement 


Nr. 


Experimental 

% 


Control 

Nr. 


% 


Individual 


Practice 


52 


53.6% 


42 


49 . 4% 


Information 


Association 


13 


13.4 


11 


12.9 


Two- Man Partner- 


ship 


13 


13.4 


9 


10.6 


Group Practice 


14 


14.4 


15 


17.6 


Other 


4 


4.1 


4 


4.7 



JM 
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TABLE 16 



COMPARISONS OF DISTRIBUTION GROUPS IN THE EXPERIMENTAL AND CONTROL 
SAMPLES AND IN THE POPULATION ON THE BASIS OF THE TYPE OF COMMUNITY 
IN WHICH THEY PRACTICE BY B™TE 



STATE 


POPULATION 
Total # GPs 


% 


EXPERIMENTAL 
Total # GPs % 


CONTROL 
Total # GPs 


% 


Arizona 


493 


100% 


7 


100% 


5 


100% 


2 


345 


70.0 


4 


59.1 


3 


60.0 


3 


97 


19.7 


2 


28.6 


0 


— 


4 


47 


9.5 


1 


14.3 


1 


20.0 


5 


4 


• 6 


0 


— 


1 


o 

• 

o 

OJ 


California 


7,631 


100% 


30 


100% 


28 


100% 


1 


4,674 


61.8 


11 


36.7 


14 


50.0 


2 


1,923 


25.2 


9 


30.0 


8 


28.6 


3 


769 


10.1 


7 


23.3 


5 


17.9 


4 


251 


3.3 


3 


10.0 


1 


3.5 


5 


14 


.2 


0 


— 


0 


— 


Colorado 


673 


100% 


10 


100% 


8 


100% 


1 


307 


45.7 


7 


70.0 


2 


25.0 


2 


68 


10.1 


1 


10.0 


2 


25.0 


3 


143 


21.2 


2 


20.0 


0 


— 


4 


107 


15.9 


0 


— 


3 


37.5 


5 


48 


7.1 


0 


— 


1 


12.5 


Idaho 


273 


100% 


6 


100% 


5 


100% 


2 


27 


9.9 


2 


33.3 


0 


— 


3 


60 


22.0 


1 


16.7 


0 


— 


4 


150 


54.9 


3 


50.0 


5 


100.0 


5 


36 


13.2 


0 


— 


0 


— 


Montana 


258 


100% 


6 


100% 


5 


100% 


2 


35 


13.4 


1 


16.7 


0 


— 


3 


28 


10.9 


0 


— 


0 


— 


4 


144 


55.8 


3 


50.0 


3 


60.0 


5 


51 


19.7 


2 


33.3 


2 


40.0 


Nevada 


119 


100% 


2 


100% 


1 


100% 


2 


83 


69.7 


1 


50.0 


0 


— 


3 


6 


5.0 


0 


— 


0 


— 


4 


22 


18.5 


1 


50.0 


1 


100 


5 


8 


6.8 


0 


-- 


0 


— 
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TABLE 16 (continued ) 



i 

! 

I STATE POPULATION EXPERIMENTAL CONTROL 





Total # GPs 


% 


Total # GPs 


% 


Total # GPs 


% 


I New Mexico 


213 


100% 


4 


100% 


3 


100% 


2 


48 


22.5 


1 


25 


0 


— 


3 


36 


16.9 


1 


25 


1 


33.3 


1 4 


110 


51.7 


2 


50 


2 


66.6 


l 5 


19 


8.9 


0 


— 


0 


— 


j Oregon 


698 


100% 


10 


100% 


9 


100% 


1 2 


324 


46.4 


4 


40 


2 


22.2 


3 


160 


22.9 


3 


30 


5 


55.6 


4 


200 


28.7 


2 


20 


1 


11.1 


5 


14 


2.0 


1 


10 


1 


, 11.1 


Utah 


293 


100% 


6 


100% 


4 


100% 


2 


191 


65.2 


3 


50 


1 


25 


3 


56 


19.1 


1 


17 


0 


_ _ 


1 4 


28 


9.6 


1 


17 


3 


75 


1 5 


18 


6.1 


1 


16 


0 


— 


Washington 


1,162 


100% 


17 


100% 


14 


100% 


t 1 


473 


40.7 


6 


35.3 


6 


42.9 


2 


247 


21.3 


5 


29.4 


5 


35.7 


1 i 3 


167 


14.4 


3 


17.7 


2 


14.3 


t 4 


264 


22.7 


2 


11.8 


1 


7.1 


I 


11 


.9 


1 


5.8 


0 


— 


Wyoming 


154 


100% 


2 


100% 


2 


100% 




130 


84.4 


2 


100 


2 


100 


l 5 


24 


15.6 


0 


— 


0 


— 



I 

l 

I 
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TABLE 17 



COMPARATIVE DISTRIBUTION OF GPs IN THE EXPERIMENTAL AND CONTROL SAMPLES 
AND IN THE POPULATION ON AN URBAN, SEMI- URBAN, AND RURAL BASIS 



TYPE OF AREA 


POPULATION 
NR. GPs % 


EXPERIMENTAL 
NR. GPs % 


CONTROL 
NR. GPs % 


GROUP 1 


5454 


45 . 6% 


24 


24% 


22 


25.9% 


GROUP 2 


3291 


27.5 


31 


31 


21 


24.7 


GROUP 3 


1522 


12.7 


20 


20 


14 


16.5 


GROUP 4 


1453 


12.1 


20 


20 


23 


27.0 


GROUP 5 


247 


2.1 


5 


5 


5 


5.9 


URBAN: 1 + 2 


8745 


73. 1 


55 


55 


43 


50.6 


SEMI-URBAN: 3 


1522 


12.7 


20 


20 


14 


16.5 


RURAL: 4 + 5 


1700 


14.2 


25 


25 


28 


32.9 
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J TABLE 18 



COMPARISON OF THE EXPERIMENTAL AND CONTROL GROUP PHYSICIANS 
WHO COMPLETED THE NINE CONTENT TESTS 



I 




AGE 



Age 


Experimental (N = 73) 
Nr. % 


Control (N = 
Nr. 


36) 

% 


60’s 


2 


2 . 7% 


6 


16.7% 


50's 


29 


39.7 


10 


27.8 


40's 


27 


37.0 


13 


36.1 


30’s 


15 


20.6 


7 


19.4 


TABLE 19 


1, 


TYPE OF PRACTICE ARRANGEMENT 






Experimental (N=73) 


Control (N = 


36) 


Type 


Nr. 


% 


Nr. 


% 


Individual Practice 


37 


50.7% 


14 


38.8% 


Informal Association 9 


12.3 


7 


19.5 


Two- Man Partnership 12 


16.4 


4 


11.1 


Group Practice 


11 


15.1 


7 


19.5 


Other 


4 


5.5 


3 


8.3 


No response 


— — — 




— — ~ 





TABLE 20 



NUMBER OF HOURS WORKED PER DAY 




Experimental(N = 73) 


Control (N = 


36) 


Hours 


Nr. 


% 


Nr. 


% 


0-5 


1 


1 . 4% 


1 


2.8% 


6-8 


14 


19.2 


6 


16.7 


9-11 


42 


57.4 


23 


63.8 


12 - 14 


13 


17.8 


6 


16.7 


15 + 


3 


4.2 


0 


— — — “ 






TABLE 21 






DATE OF GRADUATION 




Experimental (N = 73) 


Control i'n = 


36) 


Grad. Date 


Nr. 


% 


Nr. 


% 


1960’s 


4 


5.5% 


4 


11.1% 


1950’s 


27 


37.0 


16 


44.4 


1940’s 


26 


35.6 


6 


16.7 


1930’s 


16 


21.9 


10 


27.8 






TABLE 2 2 



MILES FROM MEDICAL SCHOOL 



Miles 


Experimental (N - 73) 
Nr. % 


Control (N = 
Nr. 


« 36) 

% 


0-9 


12 


16.4% 


4 


11.1% 


10 - 24 


9 


12.3 


8 


22.2 


25 - 49 


10 


13.7 


1 


2.8 


50 + 


42 


59.6 


23 


63.9 



Considerations of age, type of practice arrangement, length of workday, 



date of graduation from medical school, and distance of office from medical 
school were studied to see if the experimental group differed from the control 
group significantly in any of these aspects. Chi-square values were computed 
where feasible and showed no significant difference in these factors. 

TABLE 23 



Date of Graduation Date of Graduation 

Control vs . Experimental (completers) Control vs . Experimental (non-completers) 
k + 4.784 (not significant) x 2 = 8.647 (not significant) 

Type of Practice Arrangement Type of Practice Arrangement 

Control vs. Experimental (completers) Control vs. Experimental (non-completers) 

x 2 ~ 2.228 (not significant) x Z = 1 .978 (not significant) 



The conclusion is therefore that the presentation of the experimental 
variable, the teaching program itself, accounts for the high degree of signi- 
ficant difference between the experimental and the control groups. The only 
additional comment concerns the obvious fact that one would expect this 
difference in content learning no matter what kind of teaching program were 
applied. The point here is that the differences are of such order as to conclude 
that this program not only imparted information, as all programs might do, but 
that it imparted information in a highly successful manner. 

One further comment needs to be made on the nature of the physicians in 
the experimental group who left the study: they gave reasons which are listed 

/if 
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below. "Too busy" occurs frequently as does "machine problems," as will be 
seen later; these comments are also typical of those who stayed in the Project. 
Nevertheless, it is important to note the kinds of factors which cause a physi- 
cian to put aside something of great potential use to him. 

For example if MIP is a prototype of programs which will save physicians 
considerable time, the complaint "too busy" is somewhat difficult to compre- 
hend. Of course, the physician may not perceive MIP as helpful and he may 
also focus his attention on the time required in completion of questionnaires, 
tests and the like. 

TABLE 24 

SOME REASONS GIVEN FOR DROPPING OUT NUMBER GIVING REASON 

Too busy 7 

Machine problems and delays in receipt 5 

of programs 

No longer in general practice 2 

Deceased (1), Illness (1), Moved away (?) 4 

. Never "got around to completing forms" 4 

One participant was averse to participating in any government- sponsored 
project. There were several who stated that they would participate, but never 
did, even though follow-up phone calls elicited statements of a very positive 
nature toward the program. 

There is nothing in any of these statements to indicate anything un- 
expected; attrition of this sort is not unusual; however, a very interesting 
study might be made of what is behind a blanket statement of "too busy to 
participate." 

Acceptability 

In discussing the subject of acceptability, first are considered the first 





nine programs as in the section on content learning; then there are comments 



on the other six programs (numbers 10-15) on which fewer data were collected 
Since physicians in private practice are free agents in the manner in 
which they choose to continue their own professional education, it is not 
enough to note that a highly efficient teaching program has been devised. It 
is conceivable that a highly effective teaching program might still contain 
factors which would militate against its use in the future by these same men. 
Therefore, after having established the efficiency of the program, it is the 
next order of business to assess as realistically as possible, the probability 
of such procedures gaining acceptance with a significant number of physi- 
cians as a means of continuing their own medical education. One aspect 
of measuring this ultimate acceptance is to attempt to measure their accep - 
tance of this particular project. Another is to attempt to assess their 
preferences in continuing education and problems which they encountered 
in attempting to continue professional training by means other than those of 
the Project. Questions of this latter type were explored by the Pre and Post- 
Project Questionnaire administered to the experimental group. 

No attempt was made to treat these data except descriptively. As can 
be seen from Table 25, there is no trend indicating change from pre-to-post- 
experimental conditions. What emerges instead is a description of the 
activities of these physicians with respect to formal and informal attempts 
to their part to continue postgraduate medical education. 

On the Post- Project Questionnaire, over 89% of the physicians respond 
iny found "keeping up" with medical advances moderately or v*ry difficult;? 7 




found the reading of journal articles too time-consuming in terms of facts 
gained; about 10% found the reading of Journal articles relatively easy; the 
other physicians felt that journal articles were moderately frustrating in terms 
of facts gained versus time expended . 

On the other hand, the general experience with postgraduate courses was 
not so troublesome. Less than 4% felt that "they could more profitably spend 
their time elsewhere" than in postgraduate courses. Of ten problem areas 
discussed, in terms of percentages of physicians who felt they shared these 
problems, the following were the most troublesome to them: (1) reading of 
journal articles: (2) time loss from their families; (3) large number of patients 
consuming a great deal of their time. Finally, the following are the areas 
which did not present as much of a problem: (1) availability off relevant materi- 
als; (2) time consumed by hospital staff meetings; (3) general quality of post- 
graduate courses. In between these extremes, were these areas: (1) avail- 
ability of fill-in M.D.'s to cover for the time when the physician is busy with 
continuing education; (2) tape recordings which are too lengthy; (3) scheduling 
problems for the educational opportunities which they are; (4) lack of ap- 
propriateness of material available. 

Speaking generally, what emerges is a portrait of a busy doctor, with 
too many patients , too little time for personal use and exhibiting impatience 
with the use of journals. However, he does not feel that meetings at the 
hospital are a great problem, nor is he highly critical of postgraduate courses 
available to him. If he were less busy and the material could be presented 
in a less time-consuming way, his needs would be better met. Significantly, 
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for my practice. Moderately appropriate 35 36.1 33 42.85 

Not at all appropriate 52 53.6 40 51.94 
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of 14 professional periodicals generally available, and using the criterion 
"I read parts of every issue, " only four periodicals were mentioned by 40% 
of the sample. That is, only 40% felt that they could make that statement 
about four of fourteen journals. 

When asked questions about the quality of the MIP methods as com- 
pared to other methods (see Table 26, question 9), the MIP approach was 
preferred by as few as 54.5% (Program III) and as many as 80.0% (Program II) 
If we allow some "suggested changes, " the percentages range even higher 
(70.2% in Program IV to 90.4% in Program II). Classes and seminars were 
preferred by a range of 3.2% (Program II) to a high of 14.1% (Program VIII). 
For conventional film or television programs, the range was 3.2% for Program 
II to 15.7% for Program I. On verbal statement alone, the MIP is heavily 
preferred in all 9 programs. 

Overall ratings for this material were very high as indicated in the 
answers to question 11 in Table 26. Further, as can be seen from the 
responses to questions 1 through 7, there is very little comment of a really 
negative type on the 9 programs. Only a few comments need be made, since 
the Table is self-explanatory. 

(1) Only programs VI and IX seemed to satisfy fully the criterion 
of "newness and usefulness" (question 3), yet only Programs 
IV and V were rated low in this category by more than 20% of 
the sample. 

(2) Every program except III was rated high in "general importance" 
(question 2); only Programs II, III, and IV were not rated high 
in "personal importance" by a majority of the sample. This 
pattern seems to indicate special problems with Program III 
itself. 
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The overall Impression Is that all except III were at least satis- 
factory in "content" to a majority of the sample. 

(3) Questions 5,6, and 7 Indicate generally satisfactory or high 
ratings on the technique itself again In all programs except 
Program III. Program III certainly needs special study to as- 
certain the nature of the problem . 

(4) Finally, an overall rating of "very good" or "excellent" was given 
to all nine programs by a range of 66.3% (Program V) to 95.1% 
(Program Vj) , Only Program V was rated as "not very good" by 
over 10% of the sample. 

The conclusion Is that MIP Programs made a very favorable Impression 
on the sample and that other sources of Information are rated much lower to 
MIP. Even when one considers the heavy conditioning of physicians to mono- 
graphs and journal articles as sources of basic information, the time-consum- 
ing aspects of journal reading and the superior technique of MIP can be 
shown in this sample. In fact, the most favorable reaction to another means 
of presentation than MIP was the 15.7% preferring films or television and 13.5% 
preferring classes, seminars, etc,, in Program I, while 68.6% preferred MIP. 

Clearly, MIP has made a very favorable impression on this sample and 
this fact, coupled with the statistics concerning content learning, makes MIP 
approaches very difficult to minimize in terms of the potential for physician 
education. When one considers this with the crucial problems of time saved 
and convenience, it is necessary to conclude that the first objective (p.9) 
has been met completely. 

Now to turn briefly to the other six programs (Table 27), we find the 
number of respondents ranging from 71 down to 60. In these programs, and con- 
sidering content, we continue to find generally high or medium ratings for all 
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programs; only Program XIII had over 10% rating the content low in personal 
importance, and only Program X had 10% rating it low in newness and usefulness. 
Ratings on "organization" and "audio" remain fairly high; only Program XI had 
over 10% rating it low in visuals. "Attention Value" reflects a larger low rating, 
with all of the six programs receiving 15% or more low rating. When one moves 
on to "preferred sources of information, " MIP with or without changes is still 
the choice for all six programs. However, Program XI received a full 30% who 
would have preferred classes or seminars to MIP. Other programs which had 
o’^er 10% preferring classes and seminars were Numbers X through XIV. On 
Program XI, only 10% would have preferred films. Incidentally, Program XI was 
the only one of the fifteen on which less than 60% preferred MIP. 

Since all fifteen programs were rated on question 9 for MIP as is or MIP 
with changes, the section later in this report dealing with suggestions for 
improvement is most relevant, since as can be seen from Tables 26 and 27 , 
a large number of physicians preferred MIP only with the stipulation that certain 
changes be made. The section on positive and negative criticisms augments 
these ideas. 

A 

The overall rating of the last six programs was generally high, with a 
low of 70% rating Number XI as "very good" or "excellent" to a high of 86.6% 
giving the same rating to Number XIV. The analysis of these data is limited to 
these statements, since the information on content learning for the last six 
programs did not include data on any control group. 

Figure 23 summarizes the ratings of the various information sources for all 
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fifteen programs. 

Reason for Participating in MIP 

Table 28 lists the reasons given for participation by a sample of 50 inter- 
viewed on this subject. The figures exceed 100% since the physicians often 
gave more than one reason. 



TABLE 28 



Reasons for Participating in MIP (From Interview) 




REASONS 


N = 50 
TOTAL 


PERCENTAGE 


1. Intrigued with technique 


34 


68% 


2 . Need for information 


30 


60% 


3 . Convenience of. time and place 


20 


40% 


4. Sponsored by med. school 


6 


12% 


5. Time involved 


5 


10% 


6. No cost 


3 


6% 


7. Other 


12 


24% 



As can be seen from this table, need, convenience and interest in the 
technique or concept were the most important factors involved. These three 
items certainly describe the conditions of importance ir. a project such as 
MIP; one might expect the curiosity factor to diminish in import? ace as time 
went on and familiarity with technique increases. Still, the need and con- 
venience factors are of first order of importance. 

Utilization Survey 

Three questions were asked of participants concerning the utilization of 
the MIP system with colleagues. These questions and responses are summari- 





zed in Table 29 . 



TABLE 29 



UTILIZATION SURVEY - RESPONSES 



1 . How many people were involved or how often was the MIP system used for 
these other purposes? 

a. 3-5 times 

b. 10-15 people; 4 students at LPN school monthly 

c. About 10 

d. 6 interns training programs once a year 

e . 2 times 

f. I plan to loan entire set to associate when complete 

g. 2 times 

h. Used by office staff 

i. 3-4 M.D.'s not often 

j. Occasionally - partial viewing by colleagues and nurses 

k. 6 people 

l. Used illustrations of newborn for High School class to illustrate 
newborn's needs, dependancy and responsible parenthood. 

m. 12 AAGP members 

n. 2 times to two people 

o. 2 times to eight members of the ski patrol 

p. 14 nurses, 6 colleagues, 3 allied medical personnel 

q. Resuscitation 2 times to Fire Dept. . . "Big Event"; 5 colleagues 
thought very good . 

r. Program #1 - 60 to 80 times hundreds of people; others as noted 
20 nurses; 6 doctors 

s . One or two colleagues 

t. Small number internally at their leisure 

u. . Accompanied lecture to the Army Reserve Staff of my hospital 

v. 2 colleagues on 3 occasions 

w. One person (nurse) 

x. Approximately 12 people on 2 occasions (nurses) 

2. How did your colleagues respond to it? How useful did they find it, in 
your opinion ? 

a . Very useful 

b. Mostly felt it very useful 

c. G. P.'s felt it was quite useful;specialists were indifferent 

d. Excellent for small groups; they benefited from their use 

e. Used one time with good response 

f. Useful but (1) failed to take full advantage 

g . Good 

h . Liked it 

i. Found it interesting — but only view an occasional program in part 
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j . Have not had opportunity to share with colleagues , believe 
they would be favorably impressed as GP's. 

k . Excellent Response 

l. They were favorably impressed 

m . Very favorable 

n. Mixed opinion. Subject material was useful; presentation 
with filmstrips in time-wasting 

o . 3 saw irregularly- very good 

p. Agreed with me in usefulness 

q . Moderately helpful 

r . Enjoyed it 

s . Useful 

t. Quite interesting-slightly hard to see because of small 
screen (Army Reserve staff at hospital). 

u. They thought the programs were instructive and useful 

v. Moderately interesting 

If you used the system for training of allied medical personnel 

(1) How did they respond ? 

(2) How useful did you find the system as a teaching tool ? 

a. (1) Well 

(2) Quite useful 

b. (1) Enthusiastically 

(2) Very Good, could repeat until they got it 

c. (1) Enthusiastic 

(2) Excellent for nurses and allied medical personnel 

d. (1) They enjoyed them and felt they were worthwhile 
(2) Good for small groups 

e. (1) Not used 

f. (1) Well to resuscitation 

(2) I really haven’t used it adequately 

g. (1) Did not use it for this-but I believe it would be useful for this 
(2) Not used 

h. (1) Well 

(2) Helpful 

i. (1) Daughter who wants to study medicine was greatly impressed 
(2) Quite useful 

j . (1) Not used 

(2) Helped me take care of a program requirement (AAGP) 

k. (1) Well 

(2) Excellent 

l. (1) Good attention, good response 
(2) Very useful 

m. (1) Office personnel-interested and offered criticism: "Too 

technical", also "repeat showing several times"; 

(2) Very useful in my limited application. Should have used it 
more frequently for this function but. . .etc. , etc. 




n. (1) Great to #1 
(2) Excellent 

o. (1) Not used 

(2) I don't do any teaching 

p. (1) Interested 
(2) Very helpful 

q. (1) No-my viewing equipment is at residence and not available to 

office personnel 

(2) It would be an excellent teaching tool 

r. (1) With interest 
(2) Moderatelv. 



As can be seen, the entire populauon did not reply, but the replies indicate 
a wide range of uses to which it was put by the 24 respondents and suggest 
that these be studied to ascertain if some of these contexts in which they 
were used might be of interest for further study. As can also be seen, most 
of the uses were met with success. One cannot generalize at all from this, 
since it is entirely possible that only successful people would wish to 
reply to these questions. Nevertheless, there certainly is a fertile ground 
of suggestions here to follow up in any study of situations in which MIP 
might be employed as a tool in general use or in other controlled studies. 
Personal Importance 

Table. 30 indicates the reaction to the question of relative personal 
importance in terms of content. Means are given for the seven categories 
with a grand mean of 5.29. The range was from 4.43 for Number 4 to 
6.02 for Number 1. The interpretation of such a set of figures are on the 
high side of "average" (4.00 would be a score of "average" importance); 
we could not conclude that these programs are scored "high" in personal 
importance . Since each program is one special topic in the whole range 
of interests of the general practitioner, it is perhaps unrealistic to expect 
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a very large proportion of ratings of "7" since some would perceive this high 
a rating as being reserved for the most central facet of their practices . When 
one observes the spread of the figures across the seven rating categories, it 
can be seen that the distributions are all skewed on the high side with al- 
most no ratings of "2" or “l", so a median would be a better measure than 
the arithmetic mean (medians are given as the first column in the Table). A 
tentative conclusions is then that the population responding perceives these 
topics to be of high personal importance . 
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Full Participation in MIP 

Not every participant returned all programs. Tables 31 and 32 give the 
breakdown on returns. Table 31 gives the breakdown according to Urban, 
Semi-urban and Rural location of the participants. In all three categories, 
over 70% of the participants returned 10 or more programs. The sample size 
is too small to attempt to give any significance to the breakdown, except 
to say that the urban sample seemed to have the higher rate of returns . If 
this were studied on a larger scale, it might prove to be important since 
rural physicians have fewer opportunities to supplement their postgraduate 
education and it would be unfortunate if they were the ones who returned 
the fewest number of programs as the Table seems to indicate. 

TABLE 31 



Number of Programs Returned by Urban-Rural Distribution 



Number Programs 
Returned 


Urban 
(1 & 2) 


Semi- Urban 
(3) 


Rural 
(4 & 5) 


Total 




Nr. 


% 


Nr. 


% 


Nr. 


% 




0 Proarams 


5 


9.1 


2 


10.0 


1 


4.0 


8 


1 - 3 


1 


1.8 


1 


5.0 


1 


4.0 


3 


4-6 


5 


9.1 


0 


— 


1 


4.0 


6 


7-9 


4 


7.3 


1 


5.0 


3 


12 .0 


8 


10 - 12 


5 


9.1 


5 


25.0 


6 


24.0 


16 


13 - 15 


35 


63.6 


11 


55.0 


13 


52.0 


59 


TOTALS 


55 


100.0 


20 


100.0 


25 


100.0 


100 


All 15 Programs 


29 


52.7 


8 


40.0 


10 


40.0 


47 



Table 32 gives the same breakdown using age of the physicians as the 
criterion. Some of the same remarks apply here; one would guess that the 
older physicians would tend to have more need of help than the younger men 
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In keeping current; the Table Indicates that the "middle" group, the 40-49 
year old category, had the poorest returns. Again, little more can be stated, 
except to say In general, that the amount of participation was good overall 
since about 75% did most (at least 10) of the programs. 

TABLE 32 



Number of Programs Returned by Age 


Number Programs 
Returned 


Age 

30-39 40-49 50-59 


60+ 


Totals 




Nr. % Nr. % Nr. % 


Nr. % 


; T, f;... 1 o 



O Programs 


2 


11.8 


5 


10.9 


1 


2.8 


0 


— 


8 


1-3 


0 


— 


3 


6.5 


0 


— 


0 


— 


8 


4-6 


0 


— 


3 


6.5 


3 


8.6 


0 


— 


6 


7-9 


0 


— 


4 


8.7 


3 


8.6 


1 


50.0 


8 


10-12 


2 


11.8 


8 


17.4 


5 


14.3 


1 


50.0 


16 


13-15 


13 


76.4 


23 


50.0 


23 


65.7 


0 


— 


59 



TOTALS 


17 


100.0 


46 


35 


100^0 


2 


100.0 100 


All 15 Programs 


12 


70.6 


16 


34.8 19 


54.3 


0 


47 



Summary 

It seems justified to say that criteria cf acceptability are met by our sample. 
The preceding discussion Indicates strong approval of MIP, preference for it 
over other forms of postgraduate medical education, and a range of construc- 
tive suggestions for Improvement which Indicate a high degree of Interest. 

Further studies, comparing effectiveness of various other means of trans- 
mitting Information are certainly suggested. However, It seems safe to 
say that with Improved equipment performance and some further checking by 
previously mentioned panels to Insure the proper level of training, MIP 
would be very hard to better In effectiveness and acceptability. 
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Communications Behavior Change 

This is undoubtedly the most difficult of the areas studied to assess 
adequately within the confines of the study . Table 25 summarizes the data 
collected pertaining to these questions. Did members of the sample signifi- 
cantly change their communications behavior as a result of their experience 
with MIP? This question can not be answered definitively. 

A few general comments need to be made at this point. Behavior 
change in any experienced person in a well structured profession must be 
judged to be difficult to attain. The study as constructed had other objec- 
tives which seem to be adequately met, as has been discussed in previous 
sections on content learning and acceptability. In view of (1) the fact that 
the physicians indicate that they accept MIP and (2) the additional fact that 
the test scores indicate they learned from the programs, the idea that they 
would not or did not change their communications behavior as a result of 
this experience is difficult to explain. However, one important aspect of 
any study of behavior change is missing from the structure of the study: 
that is, the idea that it is realistic to assume that expectation of behavior 
change is realistic in the first place. Since MIP-style instruction is not 
part of the "real world" of the physicians' communication efforts, change in 
behavior is not realistic. That is, films, film strips, journals, monographs 
etc. , are the "real world" of the physicians' attempt to improve his medical 
knowledge. They are available; he knows where and how to obtain them; he 
has "made his peace" with them. MIP is part of an experimental research 
program and is not perceived as part of this "real world" of which we have 




been speaking. Therefore, the more direct way to measure communications 



behavior change would be to restructure this "real world" as having MIP as 
one of its "inhabitants. " That is, until the physician perceives MIP as on 
the same footing as other more conventional means of communication, it is 
impossible to measure bis behavior change with respect to it. 

The foregoing implies that structured studies need to be made to decide 
whether, behavioral change will occur if_ MIP is available on the same basis 
as other means of continuing medical education. The obvious difficulties 
here are enormous, but when one considers the whole matter from an economic 
standpoint, it might be solved if funds were available for wider dissemination 
of MIP materials in the same way as are other aids. That is, commercial 
enterprises actually selling such material could be considered and also, 
funding by foundations and government sources could be considered. Once 
MIP is part of the "real world, " a reassessment of MIP's effect on physician 
communication behavior can be made. Such was not possible within the 
range of the current study. The findings of the current study can be sum- 
marized simply in two statements. First, there is no evidence of signifi- 
cant change in physician communication behavior; second, the design 
employed to measure this area is not adequate to conclude that this is an 
important criticism of the Project. 

Summary of Commentary on Specific Questions Raised 

Before the physicians had been questioned about the various sources 
of information, they were asked, "How difficult do you find it to keep up 
with medical advances?" Very difficult? Moderately difficult? Not 




particularly difficult? 



In the pre-questionnaire, 22% found it very difficult, and 64% found it 
moderately difficult. The corresponding figures for the post-questionnaire 
were 2i% and 69%. (Information is given in Table 25). ! 

The physicians then were asked about the causes of the difficulty; how 
appropriate were various reasons for not engaging in as much continuing 
education activities as they might like to. (One of the questions in the 
questionnaire survey dealt with the relative importance or appropriateness 
of various deterrents to their engaging in continuing education activities.) 
The most important problem in the pre-questionnaire survey was the feeling 
that they were away from their own families too much already. Over three- 
fourths of the physicians ascribed appropriateness to their situation to this 
factor. Since many of the types of continuing education activities, such as 
postgraduate courses and lectures require the physician to take time out not 
only from his practice, but from his family as well, types of activities such 
as MIP may well help this problem. However, it should be noted that other 
types require a different kind of "being away" . . . i.e. if he is reading a 
journal, the physician cannot spend time with family doing family things . 
But perhaps at least less time is required and physical proximity to his 
family is retained. 

Three-fourths of the respondents felt that it took too long to wade 
through journal articles to get the facts that they need for their practice. 
This was verified in the interviews where the journals came in for a great 
deal of criticism such as" too much theory, too esoteric, contained too 
little that was of interest or of practical use to a GP." 




Sixty-eight percent before and seventy-nine per cent after felt that they 
had too many patients to care for to engage in as much continuing education 
activities as they might like to; practice load kept them too busy. Sixty- 
one per cent before and sixty-six per cent after felt that their experience 
with tapes was that they were too long and the inability to skim through the 
material they already knew was a problem in such things as Audio- Digest. 
This is interesting in light of the finding that many suggested tapes because 
of replay ability. 

Educational opportunities that were available but didn't meet their 
own schedules was considered an important deterrent by 54% before and 
61% after. These opportunities included use of medical TV in hospitals 
and postgraduate courses. 

Forty-five percent before and thirty-eight percent after had previous 
unsatisfactory experience with postgraduate courses. This apparently 
kept them from participating in more postgraduate courses. The feeling 
is that little thought was given to meeting needs of General Practitioners. 
Inadequate preparation often resulted in an experience disappointing to 
General Practitioners. 

The fact that subject matter of courses offered and other types of 
continuing education did not meet their needs for their practices deterred 
43% before and 45% after from continuing educational activities. This em- 
phasizes the need for more adequate analyses of content needs by post- 
graduate program directors. 

Lack of someone to care for the physician's patients while he might 




be away attending a postgraduate course or lecture was a deterrent to approxi- 
mately 42% before and 49% after. 

Over a third (38% before and 34% after) of the respondents reported the 
lack of availability of continuing education activities, other than the mail in 
their area, as a deterrent. This was relatively more important to physicians 
in rural than to those in urban areas. 

Anotner deterrent given by over a fourth before and over a third after was 
the fact that they had to attend so many staff meetings at their hospital that 
they did not have time for other methods. 

This inforrhation seems to indicate that the other questions asked on the 
pre- and post-q U estionnaires yield no conclusive evidence of behavioral 
change, although there are some trends indicated. For example, there appears 
to be an increase in the use of programmed instruction, audio tape recordings 
and slide or filmstrip presentations. There is not enough of a trend, nor 
is there a consistent pattern in the kinds of changes to establish any kind 
of definitive statement concerning communications behavior change. Comparing 
of the tables, especially when one considers the small number of entries in 
some categories , is enough to point out the need for further study of the 
problem of the kind indicated earlier. The very experience of being part of 
the MIP is enough to sensitize some participants to the whole problem and 
color their thinking as they fill out a questionnaire and what they put down 
cannot be interpreted as final statement about their actual communication 
behavior. 

The thing we are trying to establish (which is really out of the scope of 




the present study) is the degree to which exposure to the kinds of materials 
contained in MIP actually changes the style of self-education employed by 
the participants. If they revert back to pre-project behavior and employ 
journals, classes and seminars, etc. , in about the same proportion as before 
the study, it can be inferred that the experience was ineffective in changing 
physicians’ behavior, since it is admitted that there are many problems 
associated with obtaining information in the traditional ways (e.g. excessive 
time required to read Journal articles) . If "keeping up" is such a problem, 
and if MIP represents a new and more efficacious means (as earlier state- 
ments seem to indicate), then the post- project Physicians should show a 

change in the direction of the use of MIP materials. 

Evaluation of Positive and Negative Statements on MIP 

Included in the Appendix is a representative selection of positive and 
negative comments made by the participants to each of the fifteen programs . 
These comments serve to augment the ideas expressed by participants at the 
conclusion of the project when they were asked to indicate what aspects 
they liked best, what they liked least, and suggestions for improving MIP 
as a system. An attempt to comment on important aspects of these statement 
follows . 

Table 33 summarizes those aspects of MIP liked best and liked least, 
given as percentages of physicians responding. The figures exceed 100% 
since physicians often made more than one comment. 

It is always difficult to perform an evaluation of a system and materials 
of this sort, since there are so many possible contaminating factors. We 
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shall point out the possible detractions from the use of such comments and 
then make the remarks which still seem appropriate. 

First of all, it is possible that an attitude more closely identified as 
"indifference" might characterize the approach of the physicians to these 
questions. They are forced to make a "good" and "bad" qualitative judg- 
ment to the overall MIP system, when the actual attitude might have been 
indifference as stated. However, to the question, "What did you like best 
about the MIP system?" five (13%) of the respondents declined to answer 
or replied with such comments as "no complaints." To the question, " what 
did you like least?" eight (20%) declined to answer or gave such statements 
as "Can't think of anything. " This lack of response was also true of the 
specific program evaluations where the range of no responses to the open- 
ended question, "What did you think of this program and why?" ranged 
from 10% (program 8) to 35% (program 7). This lack of responses may in- 
stead reflect an attitude of indifference, although there are other inter- 
pretations: ( 8 ) those who did not comment liked MIP as it was, (b) they dis- 
liked the entire MIP program, or (c) it merely reflects a difference in attitude 
toward paperwork requirements (i.e. , open-ended questions require more time 
to answer, and while many will place checks next to question items, some 
will not answer open-ended questions). At any rate, only comments which 
occur repeatedly can be considered as reliably indicating the collective 
point of view of the participants . 

Secondly, many physicians are concerned about "level" of training. 

Are the programs above or below their levels of competence? This may be 
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TABLE 33 



RESPONSES ON VARIOUS ASPECTS OF MIP SYSTEM 


LIKE BEST 


PERCENT 


LIKE LEAST 


PERCENT 


Convenience 


53% 


Level of Content 


42% 


Technical quality 


51% 


A-V Machine 


30% 


A-V Mode 


46% 


Choice of Content 


12% 


Ability to Review 


23% 


Technical Quality 


12% 


Choice of Content 


23% 


Related Paperwork 


9% 


Level of Content 


18% 


Lack of Feedback 


6% 


P.I. Aspects 


9% 


Time Involved 


6% 


Self-Pacing 


9% 


Irregular Spacing 


3% 


Miscellaneous 


12% 


Miscellaneous 


6% 
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the actual level of competence or it may be the level on which the physician 
perceives himself to_be. In a profession as structured as is medicine and 
with status a factor of great importance, it is understandable that physicians 
would be annoyed by a program which they perceive as below their levels 
of competence. This was a frequent comment made and one can only 
suggest a follow-up. study of the appropriateness of the level of presenta- 
tion of each program by a panel of medical personnel, including the specialty 
people, general practitioners, medical school faculty and even medical 



Finally, the program area itself may be a source of annoyance. Many 
men expressed a negative comment, for example, about OB-Gyn problems, 
since they themselves no longer were involved with that kind of work. In 
a larger urban center, this is to be expected. However, the GP in a small 
town or isolated community may still expect to see the whole range of pro- 
blems, and no difficulty would be expressed about the "area" of medicine 
involved in any particular program, as long as it was of general interest. 

A GP in a cool climate might not be appreciative of information on malaria , 
nor would a Floridian be interested in treatment of frost bite . Perhaps a study 
could be designed in which programs would be geared to the particular physi- 
cian population studied. 

With these opening remarks, the following are the patterns which the 
comments took. 

Negative Commentary 

There was a general feeling about level of training as previously noted. 



students 
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"Fit for first aid class , " "valuable for nurses , " "for sophomores in Medical 
School"; these and similar comments were frequently made. One might con- 
sider a follow-up on these men exploring possible factors which would tend 
to put them "above" the average GP in sophistication and training. 

Inappropriateness of topic and lack of general interest were comments 
frequently seen. Again, one would be interested in knowing if these physi- 
cians are urban or rural in their locations, since the lack of specialized help 
can make more medical topics seem relevant. 

Technical questions dotted the negative comments. These technical 
comments are of two kinds , those which involve the use of media and those 
which involve medicine . As for media-related criticisms , interruption in- 
volving changing of slides and records was listed; poor pronunciation of 
medical terms and poor photography were mentioned, especially where exact 
hue was important for understanding certain conditions (skin); the pace of 
presentation (too fast or slow) was cited; some criticism of graphs was made 
(either too rapidly displayed or not clear enough); machine breakdown, lack 
of sync between audio and visual were also listed. 

As for technical-medical questions, they blended into the previously 
mentioned criticism of level of presentation. We must separate two aspects 
of this criticism, however. In the one, it is a function of the level of 
sophistication of knowledge of the physician or. at least his perception of 
the level of sophistication he has attained; in the other, it is a matter of in- 
troducing a subject and then not giving enough detail. This latter criticism 
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occurred fairly often and could very easily be evaluated by the kind of panel 
mentioned earlier. That is, when level of training was discussed, there 
was discussed the possibility of a panel which could review these programs 
with respect to these criticisms. This is easier here where specifics of 
media technique and medical knowledge and technique could be discussed. 
Where one is dealing with level of sophistication, it becomes partially 
a psychological assessment of the consumer of the MIP materials. Neverthe- 
less, if such programs are to become a significant facet in postgraduate 
medical education, they have to overcome the problem of being considered 
too elementary by a significant portion of their consumers . 

In sum, the negative criticisms are of considerable importance and cer- 
tainly point out the need for further, controlled study of the relative effici- 
ency of different modes of instruction. A deeper study of the psychological 
components could be done, involving physicians who tend to see these 
programs as too elementary, but the difficulties there are great and per- 
haps one should conclude that this kind of reaction is not going to be over- 
come in all cases. The final remark might be that the "healthy skepticism" 
reflected in these views has other values in physician personality and should 
be left undisturbed. The kind of doctor who feels this way may have certain 
strengths * related to quality of patient care which we would not like to lose 
in our society. 

Positive Commentary 

In general, the positive commentary offsets the negative in many areas; 
also, one might say that the positive commentary on each individual pro- 
gram was somewhat less specific than the negative. In giving praise, a 
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blanket "good" Is often used without clarification. However, when analyzed 
in conjunction with the statements concerning beat aspects of the MIP system, 
there were positive comments specific enough and frequent enough to mention. 

(a) Utility is an important consideration. The convenience of the MIP 
approach, the ease of its use, the fact that relatively little time is consumed, 
and the fact that participants choose the time and place for Viewing are all 
important. 

(b) Organization and technical quality are frequently mentioned. That is, 
the material is well organized for the maximum benefit to the recipients . This , 
of course , is a function of the blending of good media technique and the sound 
medical knowledge of the consultants. The citing of organization .'s perhaps 
also a function of the fact that the perception of this kind of factor in a 
rather good indicator of the perception of this kind of factor is a rather good 
indicator of re-leaming or completion of learning in an area . The fact that 

a physician perceives a program as organized may mean that he has really 
learned the area under consideration for the first time, or that a set of con- 
cepts which had been lost to him was now revitalized. This is a central 
experience in the learning of any area studied, that one comes to see the 
area as "organized" when it previously had not been. Without developing 
a lenghty thesis, one can consider this as one of the most important kinds 
of positive comments which could be made about any teaching program. Terms 
such as "clarified," "corrected knowledge, " "good review, " "reinforces and 
renews, " are used in conjunction with the idea of organization of materials. 




(c) Audio- visual mode of presentation is considered a valuable method of 



Instruction. Many stated that the combination of audio with visuals helped 
their understanding of subject matter and improved their learning. Others 
commented that this reinforcement of audio and visuals allowed faster acquisi- 
tion of concepts and made It easier to retain the Information. Many stated 
that this method afforded them the ability to review the material quickly and 
as often as necessary for learning of concepts. 

(d) Need Is also stressed. Many men stated that they "needed" this 
Information and were glad to have It In such a convenient 2nd condensed form 
Many stated the practicality of the Information — they use It "every day. " 

Some stated that they realized how little they knew about certain conditions 
and were glad to be Informed or, In some cases, to have knowledge "re- 
freshed" since they had forgotten a great deal about a subject or had come to 
take certain things for granted. A few also stated that the programs provided 
"reassurance" that they were doing what was currently accepted as good 
medical practice. 

(e) General Importance : Even some physicians who are not currently 
working In a certain area (e.g. , obstetrics Is not done by many of the 
commentators) felt that the programs were worthwhile because of the general 
interest In the topic considered. This Is Important, since It would be easy, 
especially If the participant perceived the program as poor In any Important 
respect, to use lack of relevance as a, central criticism. For a program out- 
side the area of a physician's practice to receive the comment that the pro- 
gram was of general Interest to everyone Is high praise. 
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(f) Other Comments : Several comments involve the statement by the 
physician that he is bringing the program or the information contained in the 
program to his associates, other physicians or nurses and interns in the 
hospital. Ideally, one would see the dissemination of information to others 
not having access to the program as a criterion of its importance. 

General: The material is summarized in the Appendices with a represen- 
tative sample of the positive and negative comments for each of the fifteen 
programs given, together with the overall comments on likes and dislikes 
of various aspects of the MIP system and materials. The evaluation just 
given represents the major areas of concern in the comments. Since the pre- 
vious section on overall evaluation of the programs was very high on the 
positive side the comments serve to fill out the reasoning behind this generally 
positive reaction. The selected comments in the Appendix reflext approxi- 
mately an equal number of negative and positive , although the positive com- 
ments usually outnumbered the negative . It is important to know what these 
negative items are so that suggestions can be made for improvement and also 
for more controlled studies for the future; one must do this, however, in the 
light of the generally favorable reaction given. 

Suggestions for Improvement 

Table 34 is a summary of the suggestions for improvement in the MIP 
system and materials, remembering that the general reaction was favorable to 
the system. The table indicates that one educational area, namely level/ 
treatment of content, and three technical areas were most often cited. These 
three areas are (1) improved A-V machine, (2) method of distribution, and 
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(3) technical quality of the programs . More trouble was encountered with 
the machine than one v\ould expect. Over one-third of the participants 
had some trouble with their machines during the program-delivery phase of 
the project. Better quality control and reliability of the machines would be 
a first-order recommendation for any further studies of this kind of program 
or any ongoing educational program based on MIP. The "level" problem 
was discussed earlier in this section. 

The suggestions on the method of distribution are worth mentioning. 
Most of the participants dealing with this aspect indicated that there should 
be an adequate supply of programs available dealing with an increased 
range and variety of subject matter from which the physician could select 
the courses in which he is most interested. An index or catalog of these 
programs would have to be available to the physicians . Some suggested 
that the programs could then be available on a rental basis with an option 
to purchase, if desired, by the user. Others felt that a lending library would 
be more desirable. The equipment should be available on either a rent- 
lease basis or purchased by the physicians, if desired. 

TABLE 34 

SUGGESTIONS FOR IMPROVEMENT OF MIP SYSTEM 
AREAS OF SUGGESTIONS PER CENT 



Level/treatment of content 


42% 


Improved A -V machine 


36% 


Method of distribution 


24% 


Higher technical quality 


21% 


Choice in selection of content 


16% 


Include supplementary source material 


16% 


Miscellaneous 


6% 


Like MIP as is 


24% 
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Conclusion 



There is no need to repeat in detail all the findings of the study.. 
However, it seems clear that content learning and acceptability objectives 
have been achieved unequivocally. Communications behavior change 
requires further study. In the body of the text, several suggestions have 
already been made concerning further studies which might be done. 

One must conclude that MIP has great potential for continuing medical 
education and that efforts made to improve and augment it would be worth- 
while . With the problems of "knowledge explosion" explored at the be- 
ginning of this text and the success of MIP ir. transmitting effectively 
some of that knowledge to interested recipients, it seems relevant to 
suggest that this kind of effort be continued and expanded. 
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The Medical Information Project is still another 
attack on the serious problem of the exponential growth 
of information in medicine. 



It is an attempt to supply current information about 
a variety of medical fields to the man who feels the 
information burden the most — the general 
practitioner of medicine. 



Throughout the medical and allied health sciences 
today there is increasing concern over the problem of 
communication. All of the efforts to solve this problem — 
television, new types of continuing courses, circuit riding 
teams, etc. — have contributed something toward its 
solution, but, admittedly, the problem continues to grow. 

The Medical Information Project is a research and 
development project which is attempting to design and test 
an individualized audiovisual communcation system 
for physicians which, if it works out, will help to break 
the information barrier. 

It is not a commercial enterprise and the results will 
be available to anyone who requests them. 



For General Practitioners 



A Research and Development Project 





AV PRESENTATION 




The concept of the communication system includes 
an instrument which will sit on a desk or table in the 
doctor’s office, project colored still pictures with 
sound and allow for some form of response. 

|/ Program Content Development 

Content for the programs to be supplied the 
physicians in the sample will be developed by the faculty 
for postgraduate education of the USC School of Medicine. 
Topics for the programs were decided by the Project 
Advisory Committee listed on the front page. 

|> Program Design 

Program design is the responsibility of the 
Department of Instructional Technology of the USC 
School of Education. The general elements of the design 
process are shown in the illustration. 





The research — or feasibility testing — aspects of the 
project require the selection of a sample of 100 general 
practitioners from the eleven Western States. The sample 
will be stratified according to population and other 
requirements and then an attempt will be made to 
randomize the sample within these categories. 

Program Delivery and Evaluation 

Fifteen programs will be prepared and mailed to the 
sample at the rate of about one every three weeks for 
twelve months. Accompanying each will be (1) a program 
book to use with the audiovisual presentation, (2) a self- 
scoring content test and (3) an evaluation form. 

Mailing was begun April 17, 1968. 

✓ Research Objectives 

As indicated in the illustration, an attempt will be 
made to ascertain (1) what change, if any, occurs in the 
communication habits of the physician, (2) the nature of 
the content learning and (3) the feasibility and 
acceptability of the system. In addition to the instruments 
accompanying the programs, questionnaires and 
interviews will be used as measuring devices. 



CHANGE IN COMMUNICATIONS HABITS 
CONTENT LEARNING 
ACCEPTABILITY 
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For further information write: Dr. James D. Finn, Director 

Medical Information Project 
School of Education 
University of Southern California 
Los Angeles, California 90007 
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April, 1967 

John Doe, M.D. 

Any Address 
City, State 

The University of Southern California Schools of 
Medicine and Education are combining in a 
Research and Development project in the field 
of Medical Communication. Our general problem 
is to design a new individualized Audio-Visual 
system for the continuing education of General 
Practitioners. Our study requires a random 
sample of 100 general practitioners in the 
eleven Western states. This letter is to advise 
you that we have rolled the dice and your name 
came up. As a result, at the end of this week 
we are going to mail you a letter inviting you 
to participate in what we feel is a very inter- 
esting and exciting project and which will have, 
hopefully, great value for General Practitioners 
and medical education. Our study is not, repeat 
not, a. commercial enterprise and is funded by 
a Research and Development grant from the U.S, 
Public Health Service. 

We very much hope you can find the time to 
cooperate and that you will answer our letter 
affirmatively. Thank you very much. 

Stephen Abrahamson, 

U.S.C. School of Medicine 



James. D. Finn 

J.S.C. School of Education 
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June 30, 1967 




John Doe, M.D. 

Any Address 
City, State Zip 

Dear Dr, Doe: 

By now you have received our telegram which advised you that we would soon be 
writing you to request your cooperation in the Medical Information Project, 

As one of the 100 general practitioners selected in our random sample to 
represent the 9600 GP's in eleven Western States, your cooperation is extremely 
important to the success of the project. 

The Medical Information Project is, very simply, an attempt to invent, design 
and test a new means of communicating medical information to the general 
practitioner using an individualized audiovisual system that can be operated 
in your own office at your convenience. The project is described in some 
detail in the attached leaflet. 

We wish to emphasize that this is a project of two major divisions of the 
University of Southern California, the School of Medicine and the School of 
Education, and is funded by a research and development contract from the 
Bureau of Health Manpower of the U.S. Public Health Service. Perhaps you 
have seen a news release about the project. It is not in any way a commercial 
enterprise. 

As indicated in the leaflet, we propose beginning sometime in September 
to mail programs to you at the rate of two a month for one year. These 
programs can be played on an individual audiovisual device we will place 
in your office. The program areas have been selected by the Advisory Committee 
as described in the leaflet and will range over a wide variety of topics. 

Each will be prepared using, as a consultant, a member of the faculty of the 
U.S.C. School of Medicine. These audiovisual programs will be accompanied by a 
short (no more tUiau_-£eh minutes) test and an evaluation form. 

The other activities involved in the research pahse will be a short questionnaire 
administered at the beginning and end of the program portion of the project, and 
a diary of your communication activities (reading, meeting attendance, etc.) 
kept for four one-week periods during the 12 months the experiment is in progress. 
We also hope to interview a sample of the sample during the year. 
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April, 1968 



John Doe, M.D. 
Any Address 
City, State 



- The University of Southern California would like to 
request your help in filling out a questionnaire 
as a member of a control sample in a study we are 
doing for the U. S. Public Health Service. The ques- 
tionnaire and explanation of our study will follow 
in a few days. Thank you very much. 



L 



James D. Finn 

Medical Information Project, Director 
University of Southern California 
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CONTROL - letter and 1st questionnaire 



April 5, 1968 

John Doe, M.D. 

Any Address 
City, State 

Dear Dr. Doe: 

Several days ago we probably surprised you with a telegram asking 
for your cooperation in our study. I am enclosing with this 
letter the questionnaire we very much hope that you could find 
time to fill out and return. I am also enclosing a leaflet which 
describes our overall study. 

The project is being funded by the U.S. Public Health Service; it 
is in the field of continuing education for general practitioners, 
and is being conducted jointly by the School of Medicine and the 
School of Education of the University of Southern California. 

There are no commercial aspects to this study. 

By the use of random sapling procedures, we pulled your name as 
a representative of the population of general practitioners of 
your state and also, therefore, as comparable to our first sample 
with which our experiment is being conducted. In complex social 
research of this nature, it is very important that we establish 
the nature of a comparable control sample. By filling out this 
questionnaire, you would be helping us to do so. Thd' whole process 
should take about thirty minutes of your time, and this is all 
we need to ask of you. 

Thank you very much, and if you have any questions, please write 
me or you can reach me or my associate, Mrs. Caput, at (213) 748-2710, 
collect if you wish. 

Cordially , 

James D. Finn, Director 
Medical Information Project 
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May 8, 1968 



John Doe, M.D, 

Any Address 
City, State 

Dear Dr. Doe: 

Several weeks ago we sent you a Medical Information Project 
questionnaire with a request for your cooperation as a member 
of a control group in the project. We have not yet received your 
completed questionnaire and we are wondering if it simply slipped 
your mind. At any rate we miss your response -- it is very 
important to us. 

If you have misplaced the questionnaire, return the enclosed 
postcard and we will send you another copy. If it is still 
in your in-basket, I sincerely hope that you will take a little 
time out of your busy schedule to fill it in and return it. 

Your help and cooperation are extremely important to the study 
and will be appreciated very much. 

Sincerely yours, 

James D. Finn, Director 
Medical Information Project 
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control - letter with post-questionnaire and content test 




John Doe , M.D . 

Any Address 

Any City, State Zip 

Dear Dr, Doe: 

A year ago you were kind enough to help us out by filling out a questionnaire 
as a member of a control sample representative of the population of general 
practititioners in the eleven western states in a study we are conducting 
for the U. S # Public Health Service. 

As you may recall, the project is in the field of continuing education 
for general practitioners, and is being conducted jointly by the School 
of Medicine and the School of Education of the University of Southern 
California. We hope that the results of the study may provide information 
that will lead to the improvement of continuing education methods to meet 
the needs of the practicing physician, such as yourself. 

Our project is now coming to an end, and as part of the study we would like 
to ask that you fill out a post-questionnaire on your continuing education 
activities as a member of our control sample. Second, we have enclosed a 
test form by which we are trying to check the reliability and validity of 
the programs and tests we used during the course of the project. In field 
research of this nature, it is very important that we establish the nature 
of a comparable control sample. We hope that you can help us out this one 
last time by filling in the enclosed questionnaire and test form. Each of 
these forms should take about fifteen minutes of your time, and this is all 
we need ask of you. 

All information that you report will be kept completely confidential and 
will be used for statistical purposes only. Since our samples were selected 
by the use of random sampling techniques, the strength and validity of the 
study depends upon your help and cooperation. 

Thank you very much, and if you have any questions, please write me or you 
can reach me or my associate, Mrs. Caput, at (213) 748-2710, collect if 
you wish. 

Cordially, 



/% 



Stephen ^brahamson, Director 
Medical Information Project 
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CONTROL - follow-up to non-responders: post-questionnaire & test 




August 8, 1969 

John Doe, M. D. 

Any Address 
Any City, State 

Dear Dr . Doe : 

The evaluation of the Medical Information Project has 
begun. However, some of the essential data is missing. 

We urgently need all of the completed questionnaires and 
tests. 

If you are among the participants who have not returned 
this necessary information, we would sincerely appreciate 
its immediate return. 

If you did not receive the questionnaire and test or if 
they have been misplaced, please indicate this on the 
enclosed card. 

Cordially, 



(Mrs.) Diana Caput 
Director of Research 

DC /Id 
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Bureau of the Budget 
Approval No. 68 -R 1031 



MEDICAL INFORMATION PROJECT 

School of Education School of Medicine 

University of Southern California 
University Park 
Los Angeles, California 90007 



QUESTIONNAIRE 



All information which you report will be kept completely 
confidential in accordance with the regulations of the 
United States Public Health Service. The information 
will not be discussed with any persons outside the study 
project, and will be used for statistical purposes only. 

Your identity, nor the fact that you are participating 
in the project, will not be released or made available 
without your expressed consent. 
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University of Southern California 
University Park 
Los Angeles, California 90007 



I . GENERAL INFORMATION 
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1. What is your sex? 

Male 

Female 

2. In what year did you graduate from medical school? 



3. How far is your office from the nearest medical school? 

Less than 10 miles 

10-24 miles 

25-49 miles 

50 or more miles 

4. How far is your office from the nearest hospital or medical 
center? 



Less than 6 miles 

6-15 miles 

16-25 miles 

26 or more miles 

5. What is the average number of hours you spend EACH DAY at 
your practice (e.g., at the office, hospital, house calls, 
etc . )? 



Less than 6 hours 
6-8 hours 
9-11 hours 
12-14 hours 
15 or more hours 



ill) 



2 . 



6. Which of the following describes most closely your field of 
practice? 

(a) General practice 

(b) General practice with particular attention to a 

special area of interest 

(specify area) 

(c) Practice limited to a special area of interest 

(specify area) 

— > If you answered (b), approximately what percent of your 
patients fall within this special area of interest? 

Less than 25% 

25-49% 

50-74% 

75% or more 



7. Which of the following best describes your type of practice 
arrangement at the present time? 

Individual ("solo") practice 

Informal association with one or more physicians 

with minimum or no sharing of income or expenses 

Two-man partnership 

Group practice under some type of formal agreement; 

i.e., three or more physicians formally organized 
to provide medical consultation, diagnosis, and/or 
treatment through the joint use of equipment and 
personnel, and with income from the medical practice 
distributed in accordance with methods previously 
determined by members of the group 

Other type of practice arrangement 

(describe) 





3 . 



8. During an average WEEK of medical practice, how many hours do 
you devote to the following activities? (Please do not 
include postgraduate educational activities. These will be 
covered later in the questionnaire.) 



I 
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Activities 



a. Diagnosis and treatment of 
patients 

b. Conference with colleagues in office 
or hospital regarding your patients 

c. Travel to and from office, hospital, 
house calls, etc. 

d. Formal teaching of medical students, 
nurses, other health professionals 

e. Formal research activities 

f. Other paid medical positions 
(e.g., industrial, institutional, 
public health, etc.) 

g. Voluntary unpaid medical services 

h. Other activities (e.g., completion of 
third-party insurance forms, admin- 
istrative supervision of ancillary 
workers, etc.) 

Please specify other activities: 



k. 



Hours per Week 

hours 

hours 

hours 

hours 

hours 

hours 

hours 



hours 

hours 

hours 

hours 




Total hours 



9. We all have the problem of finding enough time to do every- 
thing we want to. Admittedly, estimates of distribution of 
time vary with seasons, activities, age, responsibilities, 
etc. (Management experts also say executives and 
professionals do not estimate time distribution accurately 
anyway.) Recognizing all of these hazards, would you make 
a general guess as to your time distributioii during an 
average WEEK. 



Activities 



Hours per Week 



a. Medical practice hours 

b. Continuing education activities hours 

c. Civic hours 

d. Personal - Family - Recreation hours 

e. Business (if any, other than 

medical) hours 

f . Other 



hours 





Total hours: 



5 
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10. Please indicate your participation in local, state, regional 
or national professional medical organizations during the 
past THREE YEARS. 



Name of 
organization 
(please spell out) 



Are you Hold office 

a or serve on Attended 

member? committee convention 

Yes No Yes No Yes No 



Made 

presen- 

tation 

Yes No 



Took part 
in seminar 
or panel 

Yes No 






11. To what civic and/or public service voluntary organizations 
do you belong? Do you hold office (s) or serve on any 
committee(s) of any of these organizations (e.g., school 
board, P.T.A., American Legion, Rotary, etc.)? 



Hold office or 
serve on committee 



Name of organization 
(please spell out) 



Yes No 
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II. COMMUNICATIONS DATA 



A. General Information Habits 



1. Please indicate which of the following general interest 
magazines or newspapers you read. 



Read 



Magazine or Newspaper 


All/Most 
of every 
issue 


At least 
parts of 
every 
issue 


Occasion- 
ally at 
least parts 
of some 
issues 


Little or 
none of 
any issue 


Atlantic 










Business Week 










Christian Science Monitor 










Esquire 










Fortune 










Harpers 










Holiday 










Life 










Look 










Nation 










National Geographic 










National Observer 










National Review 










New Republic 










Newsweek 










New Yorker 










New York Times 










Playboy 










Readers Digest 










Saturday Review 


I 








Sports Illustrated 










Time 










Today's Health 










TV Guide 










U.S. News & World Report 










Wall Street Journal 










Other s : 
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2. Approximately how many hours PER WEEK do you spend viewing 
television? 

hours 

What types of commercial or educational television programs do 
you watch regularly, occasionally, seldom or never? 



Watch 



Type of Program 


Regularly 


Occasionally 


Seldom or 
Never 


Musical Variety Shows 








Westerns 








Drama 








Documentar ie s 








News Reporting and 
Commentary 








Science Fiction 








Sports Events, Reporting 
and Commentary 








Game Shows 








Panel Discussions 








Mystery-Detective-Spy 








Comedy Shows 
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3. Approximately how many hours PER WEEK do you spend listening 
to radio? 



Hours Per Week 



a. 


In car 


hours 


b. 


At home 


hours 


c. 


Doctors' lounge 


hours 


d. 


At office 


hours 


e. 


Other (specify) 





hours 



4. 



About how often do you attend a movie? 

Once a week 

Once or twice a month 

Once every 2 or 3 months 

Seldom or never 
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B. Acquiring Professional Information 



1. How difficult do you find it to keep up with medical advances? 

Very difficult 

Moderately difficult 

Not particularly difficult 

2. All doctors face some problems in keeping abreast medically. 
Listed below are a number of comments made by physicians about 
why they do not engage in as much formal or informal continuing 
education activities (e.g., medical television, postgraduate 
courses, reading, etc.) as they might like to. Please 
indicate the degree to which each of the following fits your 
own situation. 

Problem Appropriateness to My Situation 

Very Moderately Not at All 

Appropriate Appropriate Appropriate 

a. Aside from the mail, 
there's nothing 

available in my area. _____ 

b. I have too many 

patients to care for. ____ 

c. There's no one to take 
care of my patients if 
1 leave to attend 
meetings, postgraduate 

courses, etc. 

d. I have to go to so 
many staff meetings at 
my hospital that I do 
not have time for 

other methods. 

e. It takes too long to 
wade through journal 
articles to get the 

facts. 

f. My past experience with 
postgraduate courses 
has convinced me that I 
could spend my time more 

profitably elsewhere. 



O 
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2. Continued 

Problem Appropriateness to My Situation 

Very Moderately Not at All 

Appropriate Appropriate Appropriate 

g. I am away from my own 
family too much as it 

is now. 

h. Tapes are too long, and 
I can' t skim through 
the material I already 

know. 

i. Educational opportunities 
(e.g., medical television, 
lectures, etc.) are 
available, but they just 

don't fit my schedule. 

j . What is available 
isn't what I need 

for my practice. 

You may have some special reasons of your own which aren't 
covered by the comments listed above and on the previous page. 
Please jot down any other problems not covered here that 
apply to your situation. 
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3. Please indicate which of the following professional publications 
you read. To which do you subscribe? 



Name of Publication 


Read 


Subscribe 

to 


All/ 

Most 

of 

Every 

Issue 


Parts 

of 

Every 

Issue 


Parts 

of 

Some 

Issues 


Little 
or None 
of Any 
Issue 


Yes 


No 


Journal of the American 
Medical Association 














G.P. 














Medical World News 














New England Journal 
of Medicine 














A.M.A. News 














Postgraduate Medicine 














Quarterly Journal 
of Medicine 














Lancet 














Archives of Internal 
Medicine 














Medical Economics 














Annals of Internal 
Medicine 














American Journal 
of Medicine 














Medicine 














Modern Medicine 














Other s : 
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I 

I 

I 

i 



I 

I 
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4. How often do you use the following general reference material? 



Use 



General Reference 


Regularly 


Occasionally 


Seldom or 
Never 


Physicians' Desk Reference 








Merck Manual 








Current Therapy 








Others: „ 
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5. 



a. 



The methods of continuing education that are available to the 
physician can be thought of according to the ways in which 
they are organized. How much time to you devote PER MONTH to 
the following methods of continuing medical education? 



Method 



Formally Organized 
Programs of Post- 
graduate Instruction : 



( 1 ) 

Indicate 
average 
number 
of hours 
per month 

Mark 0, 1, 2, 
etc . where 
appropriate 



( 2 ) 

Method 

not 

available 
to me 



Mark X where 
appropriate 



(3) 

Method not 
available to 
me — should 
be developed 
for use in 
my area 
Mark X where 
appropriate 



(1) Direct participation 
or attendance 



(2) Correspondence courses 

b. Other Instructional 

Arrangement s : 

(1) Demonstrations (ward 
rounds and clinics 
where you are the 
student) 

(2) Group discussions 
(local seminars 
and study groups) 

(3) Supervised clinical 
practice (where you 
are the student) 

(4) Regular staff meetings 
of hospitals 

(5) "Circuit Rider" 
programs 

(6) Lectures, panels and 
symposia sponsored 
by local, state, 
regional or national 
medical organiza- 
tions 
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5. Continued 



I 

I 

I 

I 

L 

£ 

i 

I 



Method 



c. Personal Contacts ; 

(1) Colleagues 

(2) Consultants 

(3) Detail Men 

d. Individual Efforts : 

(1) Reading 

(2) Listening 

(to tapes , etc . ) 

(3) Viewing (TV, 
films, etc.) 



(1) 

Indicate 
average 
number 
of hours 
per month 

Mark 0, 1, 2, 
etc. where 
appropriate 



(2) 

Method 

not 

available 
to me 



Mark X where 
appropriate 



I 

I 

I 

f ® 
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(3) 

Method not 
available to 
me — should 
be developed 
for use in 
my area 
Mark X where 
appropriate 
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6. In these communications activities, you may receive information 
from a variety of media. Will you estimate the amount of time 
PER MONTH or the number of times PER YEAR you have spent using 
the following media. 



Media 



a. Printed Media 

(1) Journals 



(2) Medical digests 



(3) Medical textbooks 



(4) Unsolicited medical 
literature (e . g . , 
pharmaceutical 
company literature) 

(5) Programed instruction 



b. Audiovisual Media 

(1) Medical radio 



(2) Medical television 



(3) Audio tape recordings 
and records (e.g., 
Audio-Digest) 

(4) Telephone services 
(e.g., Dial-a-Lecture ) 



itt 




( 1 ) 

Indicate 
average 
number 
of hours 
per month 



Mark 0, 1, 2, 
etc. where 
appropriate 



( 2 ) 

Medium 

not 

available 
to me 



Mark X where 
appropriate 



(3) 

Medium not 
available to 
me- -should 
be developed 
for use in 
my area 
Mark X where 
appropriate 
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6. Continued 



Media 



( 1 ) 

Estimate 
average 
number 
per year 



( 2 ) 

Medium 

not 

available 
to me 



Mark 0, 1, 2, Mark X where 
etc. where appropriate 
appropriate 



c. Audiovisual Media 

(1) Telephone conferences 
(e.g., 2-way tele- 
lectures) 

(2) Films 

(3) Slide or filmstrip 
presentations 

(4) Scientific exhibits 




m 



(3) 

Medium not 
availalbe to 
me --should 
be developed 
for use in 
my area 
Mark X where 
appropriate 
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7. Consider questions 5 and 6 on the previous pages. In the 
light of the practical considerations of your situation, 
please indicate the combinations of methods and media you 
think are the most effective in meeting your current 
professional needs. 

Examples : INSTRUCTIONAL METHODS MEDIA 



symposia 



films 



lectures 



television 



INSTRUCTIONAL METHODS 



MEDIA 



8. How do the sources of information you listed in question 7 
best meet your professional needs? What do you like best 
about them? 
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9. How do the sources of information you listed in question 7 
least well meet your professional needs? What do you like 
least about them? 



10. If the practical limitations of your practice were eliminated 
from the picture, what would be the ideal methods of continu- 
ing your medical education and keeping up to date? 
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11. If you had a chance to specify the topics for a continuing 
education program, which topics for new and/or refreshed 
knowledge would be most valuable to you on the basis of your 
own need in the light of your practice? Please be as specific 
as possible. Also indicate where you feel the emphasis should 
be — theory, diagnosis, treatment, etc. 



i 
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PROGRAM XIV. 
EVALUATION FORM 
General In 5; t.ruct Ions 



Please complete the. Evaluation Form 
on the next seven pages as soon as possible 
after viewing the program. It is important 
that your reactions he recorded while the 
details are still fresh in your mind. It is 
also important that your response be as 
direct and honest as ; possible . You will do 
us a disservice if you attempt, to spare our 
feelin'ps. Please an over all of the*, questions 
completely. All responses v?ill. r emu in 
anonymotu;. Please c]o not sign or otherwise 
i d e n l: if } 7 the e v a lu a t i o n f c >r i n . 






IV 

-2 



I 

I 

I 

i 

I 





i; 

[ 

t 

i 




6 



PROGRAM 3f. VALUATION FORM 

Program Number XXV: ANEMIA 

S ub ;] ect Ha tter C on t cm t 

1. How important to you was the information in the 
program? 

As a practicing physician, the information it 
c o n t a in c 6 v?a s : 

Vex* y imp or t an t 
Important 

Not particular l.y important 
Of no importance whatsoever 
Other (state) 



2 , How important would you say the information in 
the program would be to general practitioners as 
a group? 

I feel that, for general practitioners as I knov? 
them, the information in this program is;: 

Very important 
Important 

Not particularly ’ mpor taut 
_ _ Of no importance whatsoever 

Other (state) 





m 
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3. The program which I have just seen: 

Pj: e sent: a cl now and useful informaLi on . 

Presented information which, while not 

complete 3. y new, was Important and useful. 

___ Presented nov. 7 infonsiation which was 

interesting but not particularly useful.. 

Presented inf or mat ion which was new 

but neither interesting nor useful. 

_ Presented material which I already knev? 
V7c.ll and was, therefore, not useful. 

_ Other (state) 



4. If you felt the information presented was useful, 
for what specific purposes was it useful (e.g., 
diagnosis , treatment , research, etc . )? 



ERIC r\fi n 
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Pr op, ram ..benign 

We arc concerned here with how v?oll the program v;as 
organized to help you deal with the information it 
con talned . 

5. In general, I found the program; 

Easy to follov? and understand 

Fairly easy to follow and understand 

Somev7hat confused and difficult to under .stand 

Impossible to follow and understand 

__ Other (state) _ _ ^ 

6. Another ■ element of good program design is clarity, 
Hov: clear were the. explanations? Since v?e are using 
two sense modalities, in this question wo ask you 

to evaluate botli the audio and visual as to clarity. 





Audio . 

( language ) 


V .L C3 U Cl .r. O 

(pictures, 
i charts , etc _ 


Very clear ; 

c x n ] a n a t; :i. on r> e c c 1 1 e n t 






Re a s onab ly c 1 c ar ; 
c x o 1 r. n a t i on s «oo d 






C 1 ar i. t y o f p?; eg r am 
v; a s j u s t a d e c: u a l o 






Mot very clear; 
e x p 1 a n a t J. on s nee d. 
impr o v e m e ^ 1 1 






Unc 1 o a r ; c on f u s ing 







Other comments on audio? 



Other comments on visuals? 
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7. Another way to evaluate a program is to assess its 
ability to keep your attention. In the case of the 
program you have just seen and heard, v:ou!d you 
assess this quality by checking the statement that 
most nearly reflects your opinion. 

Excellent attention value; my interest 

never wa i v o r c d . 

High attention value; it was fairly easy 

to stay with the program. 

Moderate attention value; T. could take it 

or leave it . 

Little attention value; the program dragged. 

Ho attention value; I felt like turning off 

the machine or doing something else. 

Other (state) _ 



8. Would you make a general comment as to the 
pr c sen tat ion-c s the tic cjimli try of the program 
(i.c., quality of photography, case of reading 
print, quality of sound, artistic merit). 



i 

i: 
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Th q_ Pr o/u/^r:^ as Inf orrm t 1 on Source 

5. Given only one choice for a source of information., 
end taking into account the time, effort, etc. 
required of yoi] , cl i iocd;‘ cm ly one ansv?or in terms 
of the to pic jof this, spec i fie program. 

_ __ I would take this program as presented ’ 

I would take this program in this fori:- at ;if 

certain changes were made. (specify changes) 



I would prefer to read about this topic in 
j our na 1 s or t c >; t s . 

I v-ould prefer to attend a continuing educa- 
tion class, medical .society meeting or 
hospital seminar on this topic. 

I would prefer to watch a film or viev? a 
.medical television program on this topic. 



I would prefer to discuss this topic 
informally with a colleague or colleagues. 

1 v?ould prefer to discuss this topic v.cith 

a detail man . 

Other (state) ^ _ __ 






O 
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10. We would now like to have you rate these same 
seven possible sources- •\agn\u , tn terms of the 
topic; of this program only . In the space 
provided, rate each source of information on 
a 7-point scale. Seven is high and one is lov?. 
Two or more sources can be rated the same. 



High Lov? 



Source 


7 


6 


5 


k 


3 


7 . 


1 


The MTJ? program you 
have just seen 
















Read in g in j on r n a 3. s 
and texts 
















— — 

Class, society meeting 
or seminar 






1 










Film or medical 
television 
















Informal discus si on with 
colleagues 
















Discussion with 
detail man 
















Other (state) 

































o 
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12 , 



Considering the topic; and general quality of the 
program and my personal information needs 5 1 would 
rate this program on an overall basis as: 

Excellent 
_ Ver y good 

Fair 

_ _ _ Not very good 

_ Poor 

In the space belov?, please write your coirnien ts to 
the question: WHAT DID YOU THINK OF THIS PROGRAM 

AND WHY? 



No' 7 turn the page, and take the test. Remember that; both the 
evaluation form, and the test will be mailed back to the 
Medical Information Project in the envelope provided. 



O 
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TEST QUESTIONS 
Program XIV: ANEMIA 



1. The laboratory reports that the hematocrit of a 
patient is 20% and the reticulocyte count 
(uncorrected) is 5%. This patient's bone man* ov? 
is making red cells at 

(a) the normal rate 

(b) one-half the normal rate 

(c) twice the normal rate 

(d) five times the normal rate 

(e) ten times the normal rate 

2. A 68 year old woman with a negative history is 
found to have an anemia on a routine blood’ count . 
ihe PCV is 28%, the hemoglobin concentration is 
7.0 grams percent, and the corrected reticulocyte 
count is 1.0/;. Which of the following tests 
would be appropriate for this patient? 

(a) Bone marrow 

(b) Schilling's test 

(c) Intravenous urogram 

(d) Barium enema 

(e) Coombs test 
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3. A 25 year old man is found to have: a hematocrit 
of 20/c , a hemoglobin of 7.0 grains percent, and a 
corrected reticulocyte count of 8%. The physician 
obtained the history that .the patient was treated 
briefly two months before for an ear infection 
with oral chloramphenicol. Check which of the 
following statements is true: 

(a) Iron deficiency is not the cause of 
his anemia. 

(b) He does not need vitamin 11- 12. 

(c) Chloramphenicol did not produce marrow 
aplasia as a cause of the anemia. 

(d) All of the above are true. 



4 . 



A 17 year old negro girl is found to be anemic 
with a PC V of 30 A, a hemoglobin of 9.0 grams per- 
cent and a reticulocyte count of 1.5%. A sickle 
cell preparation was positive. The peripheral 
blood smear was thought to be unremarkable. The 
physic ian proper ly : 



(a) prescribes ferrous sulphate. 

(b) informs the girl and her parents that 
she has an inherited type of anemia. 

(c) advises the girl to eat a better diet. 

(d) suggests that more tests will be 
needed before a diagnosis can be made. 

(e) tells the. patient her blood count is 
normal. 
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A 72 year old wan was admitted to the hospital 
because of severe anemia . One week before ho had 
gone to a. pay sic tan because of weakness V7hic:h had 
been gradually worsening over a three month 
period. The doctor had given him a shot of an 
unknown medication and ordered a blood count. 

When the blood count was reported to be low 
several days later, the' patient was advised to go 
to the hospital. Admitting laboratory work 
showed a PoV of 16%, a hemoglobin concentration 
of 5.8 grams percent, and a corrected reticulocyte 
count of 7.0%. The peripheral blood smear showed 
anisocytosis , poxkliocy tosis , and oval macrocytes . 

What is your best explanation for' this combination 
of findings? How would you proceed with this 
patient? 
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6. A 22 year old woman entered the hospital after a | 

synocopal attack. She had been v;ell until the rnor- . ; 

ning of admission when she developed lower abdominal 
cravnping pain and Via cl vomited. The pain persisted, *■ 

she became "dizzy" and had then fainted. Physical \ 

examination showed a young woman in moderate distress j 

with abdominal pain. The pulse was 110/mi nutc. j 

There was lowsr abdominal tenderness but rectal and j 

pelvic examinations added no further information . 

The laboratory studies revealed the following: 

Memo - C or r . r e. t ic 





rev 


glob in 


county 


Admissi on 


22 


7.5 


5.0 


2 hours later 


19 




5 . 5 


4 hours later 


IS 




8.0 


6 hours later 


•16 




6.0 



Except for reticulocy tosis > the peripheral blood 
smear V7as normal. If you were this patient’s 
physician, v 'li at v/ould you advise? 

(a) Have the blood bank set up blood and 
prepare for an emergency splenectomy. 

(b) Administer vitamin B~12, folic acid, 
iron and transfuse with whole blood. 

(c) Line up blood and perform a culpocentesis . 

(d) Start her on adrenal steroids. 
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i 



i 

lr 

i; 







VI 

- 1 - 



I 

I 

! 



L 

i: 

1 



1. To what degree do you have difficulty in "keeping up" 
medically? (Information explosion, time problem) 

What is the primary nature of the difficulty? 

How does it manifest itself? 



2. Most physicicms figure out their own pattern of keeping 
up. How would you describe yours in general? 

3. If you need some information for a specific problem in 
a hurry, what do you do? Can you give us an example? 

4. Do you rely upon other physicians or health professionals 
for information? How, for what, and upon whom? (Not 
names, but kinds of specialists as Cardiologists, other 
G.P.'s, Gynecologists, Nurses, Pharmacists, etc.) 

5. There is some good and bad in all of these things. 

What has been your experience in general with (a) meetings, 

(b) postgraduate courses, lectures, etc. 

a . 

b. 

c . 



6. Can you tell us why you decide to: 

(a) Read an article or not. 

(b) Go to a meeting or not. 

(c) Listen to a detail-man or not. (Like/dislike, useful 
or not) 

(d) Read unsolicited mail or not. (What do you do with it?) 

7. What does your practice include: 



(a) Major operative surgery? 

(b) Obstetrics? 

(c) Minor office surgery? 

(d) General medicine for men, women, children?, 



8. In the questionnaire you returned a few weeks ago, you 

gave us information about your communication preferences, 
and this information will be very helpful to us. However, 
there are some things a questionnaire cannot supply. We 
are very interested, for example, in clues as to the best 
possible way to program our material. Give us an example 
of medical communication which particularly appealed to 
you? (Explain in detail.) Why did you like it? 
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9. Many people, especially medical school faculty, have 
been telling us that we must avoid any appearance of 
a so-called "cookbook" or "how to" program. On the 
other hand, many practicing doctors have indicated 
that they need more of this kind of practical infor- 
mation* How do you feel about this? 

10. What do you feel is needed more in the way of infor- 
mation: (Rank in order of preference if 2 or more 

are specified) 

(a) New medical advances. 

(b) Review of refreshed knowledge. 

(c) Theoretical knowledge. 

(d) Practical. 

11. During the last three (3) years, have you been asked to: 

(a) fill out questionnaires in connection with studies 

made by drug companies, A.M.A., other organizations, 
institutions, government agencies, etc? 

yes no approximate number 

Did you fill them all out? Did you fill some of 
them out? Why? 

jyes no approximate number 

12. Were there any particular aspects of the Medical Infor- 
mation Project that influenced you to participate? If 
yes, what were these aspects? 

(Check only if doctor volunteers one or more of the 
choices given) . 

no cost time involved 

_need for information intrigued with 

technique 



a medical school involved 
(sponsorship) 

convenience of time & place (e.g. in office when 
want to rather than at some other place) 

other 
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PHYSICIANS' COMMENTS ON OVERALL CONCEPT OF MIP SYSTEM: 

(1) Aspects best liked 

(2) Aspects least liked 

(3) Suggestions for improvement 
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What were the most useful aspects about the MIP as a system 
of acquiring information? What did you like best about it? 

Physicians Comments 



00 Clear, concise, factual. 

I could take advantage of it when I had the time, 
and in my own surroundings. 

02 It was well presented; clear, and concise. I 
liked being able to stop the machine and think 
about different points that were covered as we 
progressed through the film strip. 

03 It is nev; and different. 

There was always something important, useful in 
the information provided. 

04 1. Practical - easy to follow. 

2. Easy to fit into an extremely busy practice. 

3. Current therapy with good diagnosis and re- 
search. 

4. Not too complicated, consequently useful and 
easy to remember. 

5. Audio-visual - makes lasting impression. 

06 Selection of subjects. 

07 Its thoroughness with no wasted time. 

09 No- comment. 

10 Variety of topics covered. 

13 New information that I gained from it. 

15 Concise rapidly available and completed instructions 
on one subject. 

16 Can choose when I want to listen and view. 

Consist of both audio and visual learning. 

17 I lilced the availability of experts in all fields of 
medicine being able to present their subject matter 
together with visual material in the office and home 
at my convenience. Most of the time I was impressed 
with the clarity and presentation of the subject 
matter. It was right to the point and eliminated 
most padding ma.terial that is so prevelant in other 
media. 



No 




19 Timely subject matter; 
convenient ; 



condensed ; 
concise ; 
well-arranged ; 
et\sily understood 

20 Time involved - no necessity of meetings. 

If I didn’t care for it or material didn’t meet 
need I could skip it or not use it without walking 
out on lecturer. You can pick and choose information 
wanted. 

You can review at a glance. . This would be one thing 
that would be an advantage over tapes or reading . 

22 I thought it was a good project in that it was fast, 
clear and not a lot of equipment to keep around. 

Programs could be stored for review. 

23 It was concentrated and hit several of my learning 
processes at once. There were parts that were re- 
searched out and hit my needs nicely. 

24 Illustrations were helpful and the subject matter was 
easy to follow. 

26 1. Convenience 

2. Clarity 

3. Ready - review features 

4. Generally excellent choices of material. 

29 Brief and concise and complete enough that one feels 
the important points of that subject have been covered 
and can be grasped. May be reviewed as needed. 

Questions and answers regarding presented material. 

30 I was able to acquire the information at a place and 
at times convenient to me. 

Information I acquired easily and quickly could be 
given a quick "once-over " . 

Other matters could be repeated as needed until clear 
understanding and learning was accomplished. 

31 The combined audio-visual approach is of value. 

32 For the most part a good review of subjects covered 
and stimulated some extra reading on my part. 

Good visual and audio instruction and served as good re- 
view and stimulation for further reading and review. 
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33 One can do this at one's leisure. 

Also, one can review film strip as desired. 

34 Able to concentrate quietly and can repeat any film 
strip. All programs exceptionally well presented. 

36 It provided a good current review of the subjects 
attempted. Most of the material while usually familiar 
to me was useful, and I could replay the strips several 
times in a short period if the subject required it. 

37 Could be done at my convenience; could be stopped and 
considered when needed at any point in the presentation. 

38 It demands and gets my attention. 

3-way learning methods improve my learning. 

39 Brief, useful information which was pertinent to my 
practice. I liked the combination visual and auditory 
methods given. 

40 1. Brief 

2. Well planned 

3. To the point 

4. Available after work or on off days 

5. Good rough of subject matter. 

The MIP system is of definite value in continuing medical 
education. Is the cost of such a system reasonable? 

41 Quick, attention receiving review of some old and some 
new needed information. 

42 The visual aspect tied in with auditory explanations 
that are available on an "on call basis". 

44 . 1. It is able to present a fair amount of material 
in a short time. 

2. Could probably use this to fit into a schedule 

away from time needed for* the practice of medicine. 

46 1. Available at my own time. 

2. Can be reviewed, repeated if necessary. 

3. For my own acquisition of knowledge a combination- 
of audio and visual allows faster boning and stays 
with me longer . 

47 For the most part I liked the manner in which the sub- 
ject was organized and presented. 

I like the assurance that the presentation was 
authoritative - come from a good institution. 
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48 

49 

50 

51 



52 



54 



55 



56 



Very useful in reviewing and renewing basic informa- 
tion in regard to the subjects covered. It put in 
small packages subject material to be reviewed. 

Brief and concise. 

Left out unimportant material. 

Easy to set' up. 

I could study at my leisure and they utilized several 
methods at once - audio, visual, color, simplicity, 
reinforcement by answering workbook, then taking a 
test. 

The method of direct teaching was good - the manner of 
presentation. 

The material presented was often not interesting be- 
cause of its superficial approach or the lack of 
really demanding teaching . 

Ability when time permitted (often swiped from some- 
thing else) to sit down, organize thinking, and con- 
sider total of a subject for consideration. The 
combined audio-visual approach with student control is 
good. Subject matter not too expensive in field - 
limited to practical aspects of condition in question. 
Quality audio good; visual usually good; organization 
quite good. 

Ability to repeat - for review or to grab a specific 
point perhaps forgotten, was good. 

Coverage of one subject fairly thoroughly with visual 
r e inf orcement . 

Simplicity and attention keeping qualities . 

The material was well chosen; cut to the important 
elements and well presented. The visual aspect makes 
this information better retained. I could review if in 
doubt. Our anaesthetists were impressed with the 
inhalation lessen. I used the birth control unit many 
times. 

I found the programs - interesting and most of them 
well chosen. 

The presentations were well presented. Audio and 
visual were excellent. 

Many were more of review but we need that. 
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57 

58 
60 
61 



63 



64 



66 



67 



68 

69 

70 



71 



Could do at home — little time involved — no 
traveling . 

Visual part . 

Available . 

Audio-visual presentation 
Can use at own convenience 
Not too long 
Good presentation 

Can be used at my convenience 
Fairly up to date 
Fairly practical 

I could use the system at my convenience - could 
repeat records and film strips. 

Subject material was selected quite well. 

Generally speaking it presented a lot of factual in- 
formation in a short time and maintained my interest 
quite well - (changing records etc., keep me awake) . 

-Time to utilize the project could be scheduled at 
my convenience. 

-Material is condensed to most practical content with 
elimination of extraneous material. 

-The combination of audio-visual methods appear to 
reinforce memorization better than just visual. 

None 

Combined visual and oral media. 

Brevity, stress of most important points; ease of 
taking time for minute of concentrated review - a 
busy GP has to compromise in the demand that he 
know all things about all things; otherwise, he would 
become a most frenzied and frustrated individual. 

New information in limited amounts (could have used a 
bit more) presented quite simply and effectively. 

Not very time-consuming. 

It most nearly fit the audio and discussion aspects 
of personal contact. 
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73 . It was somewhat a forced program v?hich was good. 

In general, presentations were good. 

74 I was acquainted with some of the materials - but the. 
improvements and refinements in techniques over the 
past 15 years was either new or I had forgotten some 
of the material. I especially like the review 
characteristics of the things and subjects I use 
constantly. I was able to use several HIP lessons 
with nurses and at hospital staff meetings - they 
were well liked by both groups. 

77 Clearness and concise presentation . 

78 Simple, concise, and makes use of both audio and 
visual senses. 

79 Current 
Practical 
Clarity 
Useful now 

80 Concise programs ~ can choose time to study them. 
Visual as well as oral programs help in understanding. 
Very informative and timely. 

Once a week programs such as these would be very 
useful . 

81 Combined audio-visual teaching. 

82 Choice of time to devote; single topic covered. 

The requirement to participate, i.e. respond to 
questions . 

83 1. Practical programs 

2. Mostly clearly presented 

3. Good comprehension 

84 Audio-visual approach. 

85 A combination of seeing (visual), hearing (audio and 
memory training, answering questions). 

Leave a more lasting impression - could be programmed 
at my own convenience and repeated. 

Information seemed authoritative and well organised. 

86 Combination of audio and visual education. 

Easily utilized. 

Can be viewed at any time. 




U/ 



Well presented with practical information? could be 
used at convenient time; 

offered method of folloxv- up study and could be re- 
ferred to if needed at later date - eliminated need 
for notes etc. 

It forced me to spend some time each month. 

Very little. 

I lihe having the slides as well as the lecture. 

No complaints. 

Available, re-viewable , concise and good material 
(usually) . 

1. Consider it at my leisure. 

2. Usually very informative, concise. 

3. Easily reviewed. 

4. Although I did not use it thus, it could be used 
for patient education. 

This is an excellent educational technique. I 
lihed the fact that the subjects are presented both 
audibly and visually. 

It was a good review which could be utilized during 
my free time . 

Ease of use- 

Material was of value in general practice. 
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What were the least useful aspects of the MIP system? 
What did you like l-ask about it? 

Physicians Comment s 



00 I had no choice of material or subject matter. 

02 I disliked having to change the film strips and 
records so often. 

03 None really - I like it. 

04 I actually didn't look for. any non~useful aspects. 

The material - audio and visual, and presentation - 
were superb and to be commended. 

If a doctor wants further details he knows what he 
is looking for and can write for more information 
or look it up. The incentive is there after pro- 
gram is completed, and the subject discussed. 

06 Too impersonal. 

Time inadequate for subjects selected. 

Too didactic. 

07 Can't think of any. 

09 No comment . 

10 Paper work in connection with strip questions. 

13 Graphs 

15 Hauling machine around. 

Storing machine. 

Filling out critique. 

This booklet. 

16 Cannot choose what subject I want to view . 

Some are very impractical. 

Some are too downgraded. 

17 One of the least useful aspects is also availability 
which is so good that there is no enforced time or 
place factor, so that in my case I tended to put off 
the lessons. The entire impetus or real availability 
is on the student or practitioner, and being human 

we tend to procrastinate when the lessons arrive. 
There should be some degree of compulsion worked 
into the program to make it advantageous to do the 
program on time. 



No 




Time-consuming . 

Limited to subject matter. 

As in any programmed information some is going to be 
not applicable or useful to you. 

No comment 

The non-professional, and rough and cx-udely done por- 
tions of it. The visual, was practically worthless 
and should have been a great source of learning. It 
was not tailored to me: that is about five per cent 
was actually useful material. The rest was merely re- 
hash of what I've been hearing in refresher courses for 
years . 



The fact that it is impossible to review a portion of 
the program - if something requires clarification 
the entire record has to be run through again. 

Turn table. 

Frequent changing of film strips and records. 
Questionnaires! ! 

It was inflexible in that I would have preferred to be 
able to replay a paragraph or phrase as many times as 
I wished, without having to go back to the beginning 
of the record. 

The iri-egularity in receiving materials. 

No comment. 

Long wait between projects. 

Presentation of subjects unrelated to my geriatric 
practice - e.g. Pediatrics and obs. 

Many of the presentations were too superficial and 
not presented on a level of instruction commensurate 
with my experience and knowledge. I felt too often 
that I was "being talked down to". The film strip 
method is difficult to adapt to demonstration which 
in certain instances is non-useful. 

In tranquilizer lecture, would have been better for 
me if brand names v. 7 ere used in discussing various 
drugs. GYN lecture more suited fox* medical students. 
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38 Time-consuming. 

39 I didn't care for the audio-visual machine. You can't 
stop the machine at any point or pick up something you 
missed without going back to the start of a record - 
this became a waste of time for me. I would like to 
have had a brief written summary to follow and keep 

as a permanent record without having to go back to 
the records and photos to pick out any small point. 

40 1. Not specific enough on some points. 

2. Too many questions about how I like the program. 

Seems like you were looking for testimonials. 

41 Although consuming moderate time it did not seem that 
things could be presented in much depth - therefore 
the most help was in review of items already understood 
but half forgotten yet recalled with review: and new 
information could not as wall be presented - to do so 
meaningfully would have required background and appli- 
cations too time-consuming for one program and too 
boring for those familiar with it. 

42 Relatively bulky apparatus, along with the necessity 
almost to take things in order from start to finish. 

44 Not long enough. 

46 1. Many aspects may be unanswered or unclear. 

2. The audio was not as descriptive or complete as 
necessary for thorough understanding. 

3. Visual unclear or not entirely appropos to audio. 

4. Much of first few programs was well known . 

47 Occasionally the program was unrealistic. For example, 
treatment of fungus infections of the toenails for 
6-12 month with griseofulvin at a cost of $15/mo.. 

My patients won't buy that kind of treatment, at least 
none have yet. 

48 None . 

49 Screen too skimpy (small) to share material with a group. 
A written lecture should accompany each strip for 
references . 

Headlines easy to pick out for reference of material. 
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50 The only troublesome aspect is the lack of opportunity 
to ask questions on the spot such as one would have in 

a lecture. By and large there were no real bad features. 

51 If it had all been very instructive the time spent would 
have been of relatively high value. Some of the infor- 
mation was so basic as to be not useful. 

52 Feeling of being possessed by a machine!.’ Not really, 
but more so than a movie, discussion group approach, 
etc. This not serious however as "1934" in only 15 
years away! Actually the machine concept used was 
good, and for once a new gadget did not require repairs! 
Basically no real important aspects I did not like. 

54 Visual picture - small. 

55 I liked least the paper work - I am fully aware that 
this goes along with such a program and would be min- 
imized in future educational filmstrips. 

56 At times it was hard to find the time during office 
hours to follow the program through without some 
interruptions - 

But as for the MIP system as given, I find no fault, 
except I would have liked to have had one on some of 
the aspects of coronary care unit and procedures. 

57 Coverage of subject, little too brief. 

58 Much information too general. 

eb Limited scope covered by program. 

61 Poor subject material. 

Very little new or practical material. 

63 Tended to be too elementary and slow moving. 
Pronunciation very poor. 

Inability to replay part of record without playing 
whole record. 

64 Carting the machine around and signing for it as govern- 
ment property. 

66 At times the color or photograph was not really good - 
I thought the charts and drawings were quite excellent. 




Superficial treatment of some subjects in an attempt to 
cover too much ground. Also much of the material was 
very elementary and should have been well known by most 
practicing physicians. 

None . 

No feedback- 

Large equipment space required. 

Shortness of programs made sufficient detail difficult 
in considering various subjects as in "Hypertension". 
Subjects not in a related sequence of topics {I realize 
the experimental nature of the project.) 

Much of the material was not new to me. I was rather 
annoyed by mispronounced words in almost all of the 
records . 

It fell during an unusually active period of ray life. 

I am always active, to my wife's pride (and disgust) 
but this past six months have been like none other of 
my life. 

Most of the information was a review and sometimes too 
general but realize a short time element involved. 

Sometimes difficult to find a quiet half hour in which 
to sit down and be able to concentrate fully. The 
courses and subject material were excellent. 

Much of material was too elementary. 

It took time (80%) on things I already knew to get 20% 
information I could use. Some of the information was 
not appropriate to ray practice. 

1. Trying to find a quiet moment to sit and listen. 

2. Machine did not always work well. 

3. Slides and spoken words did not always match up. 

I am afraid I have no objection to these programs - 
they have been most useful and informative. 

Not enough programs per month. 

Too much time spent turning records and inserting 
film strips. Difficult to repeat short areas of infor- 
mation . 




Too much time spent turning records and inserting film 
strips. Difficult to repeat short areas of information. 

Inability to choose topic; coverage of material already 
known . 

1. Sections of some programs were too elementary. 

2. Necessary interruptions during view ing some pro- 
grams broke the chain of thought, i.e., the whole 
program had to be viewed at one sitting. 

Some of the subject matter too fundamental and not 
diagnostic or treatment oriented but this was rare. 

Couldn't get "instant playback" for something one 
didn't quite follow. Means replaying entire record 
again - I felt a tape would offer better control of 
audio. Sometimes left wondering if all of the ground 
was covered in so short a program. 

I liked the program - all phases were of interest to 
me. 

Could have outdated information and could present 
already known material - no problem if material were 
frequent enough - l/month. 

Too much time spent for the amount of new material pre- 
sented - a good review. 

Time consuming and really provided very little infor- 
mation thcit I was not already familiar with. If the 
course had been designed to add knowledge into what we 
already knew the course would have been excellent - 
these courses seemed to give only basic information. 

I really enjoyed all of it.'! 

None written. 

Can't ask it qxiestions. 

1. Probably expensive machine. 

2. These programs were usually too limited for the 
subject . 

3. Authority of the program not identified. 

I don't have any objections to program. 
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97 Little new material presented. 

Hard to go back to hear something over. 

99 Use of "brand"names together with generic names. 

Example, match tranquilizers. I never was able to 
learn names of all drugs mentioned. 
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What specific- recommendations would you suggest for improving 
the Medical Information Project audio visual system? 

Physicians Comments 



No. 

00 If programs could be prepared and catalogued so that I 
could obtain information most useful and interesting to 
me it would be a great improvement. 

02 Combining the film and sound such as a movie projector 
(cassette etc.) but keeping the feature of being able 
to stop the film and sound at any point. Also - if the 
material could, be put on a cassette for use while 
driving (without the visual portion) "double duty" could 
be made of this system. 

03 It could be a little more specific about clinical appli- 
cations in some cases. 

A different device utilizing a filmstrip and L-P record 
so you wouldn't have to change records and strips so 
often. 

I'm sure that if I had to buy such a device I would think 
2-3x about the cost in relation to the value of the infor- 
mation received. 

04 More detail in some fields - as blood gases and treatment 
in the program inhalation therapy. 

Y7ould not want any change in the presentation and audio- 
visual and material used - only more detail in some sub- 
jects . 

When these were presented to hospital staff - interns 
. and nurses - excellent response was obtained. 

06 The visual portion could be just as well presented by 
pictures on cards or in pamphlets. 

The audio portion would be more acceptable in easy to 
read pamphlet form. 

07 None. 

09 None. 

10 Longer presentation in terms of time and depth with no 
required responses and testings to be returned. 

13 Condense it and put more practical material in it. 

15 No comment. 







Make subjects more difficult - or deeper. 

Give choice of material desired - or 

Send questionnaire in advance and pick subject most 

desired by most participants. 

1. Build in some incentive or reward feature that would 
tend to compel to a non-offensive degree the prac- 
titioner to study the lesson soon after it arrives. 

1. Shorte r elective programs; 

2. Increased range, variety, choice of subject matter. 

One thing that I could think of would be, programs that 
would amplify a given subject if you were more inter- 
ested in it. This could be done in tape programs. I 
believe they were very good. Note comments in eleven. 

I feel the project is good and adequate as is. 

Have it done 100% by good professional-rnedical per- 
sonnel!!! Study us a little more. You fed me a lot of 
stuff but never did you ask me what I knew and what I 
wanted to know. I would not buy this program nor would 
I whole-heartedly recommend it to my colleagues. It 
behooves you to find out why I wouldn't. This of course 
you should have done before you started the course: lira 

not going to buy unless you have something of interest 
for me and I'm darned hard to please after twenty five 
years of exposure. The machine is miserable with its 
built-in stop-start backup, and hold pictures. My mind 
isn't a machine geared to click-clak. 

1. Some way of backing up the record or getting to a 
particular part without having to repeat the whole 
thing . 

2. Have a physician narrate or give the lecture (pre- 
sent the material). There were rather numerous 
mis-pronunciations which I do not think a physician 
would have made . 

3. Perhaps the clarity of the photography could be a 
little improved. 

4. I found it very difficult to study the Graphs and 
listen to the Speaker at the same time. 

Suggest tape audio for be tter review of this aspect - 
or redesign turn table. 
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29 More information on most of the subjects presented - 
-If possible more on each record and each film strip. 

Specifics in diagnosis and procedures in therapy are 
important to us who have been out of school for many 
years and even if we remembered now all that we were 
taught then, we would still be extremely deficient in 
the requirements of present day medical practice. Any 
medical information, be it new or a review of the old 
with specific and reliable facts is gratefully (and 
needfully) received by me. 

30 An adequate supply of programs should be available, poss- 
ibly on a rental basis, and with an option to purchase 

if desired by the user. 

Provision should be made to permit re-playing of any 
small portion of any program. 

The user should be able to lease or to buy the audio- 
visual machines. 

This could be a useful and worth-while medium for main- 
taining competence in practicing physicians. 

31 I was frustrated to the point of withdrawing from the 
program by the disorganization in getting the program 
going and the irregular supplying of the material. 

To me, material of this type should be scheduled and 
released in a predicted time interval. I began to think 
the program had been discontinued at different times due 
to the long intervals between programs. 

The lack of organization is typical of governmental par- 
ticipation in medicine in all forms. 

32 Access to full length T.V. medical programs on an indi- 
vidual pay basis (I realize that this would have to be 
subsidized) . 

Comment re Journals: AMA to which we pay substantial dues. 

Has much too many ads and these dispersed between the 
occasional practical good paper. Please advise if it's 
a fair question, what AMA really does with its tremendous 
income from all the ads it has. 

From k/ observations and experience, the GP of today is 
the one who needs the most help in trying to keep abreast 
with accepted and best medical practice. I think a T.V. 
or film program with the most common ailments should be 
available (or for hospital showings)... Topics such as 
Hypertension; Congestive Heart Failure; Allergies in 
general; Bronchial and Cardiac Asthma; and nervous ail- 
ments, especially major neurosis vs. minor psychosis, 
etc. In surgical field : from T and A's to hemorrhoids - 
also - Inguinal Hernias: Unbilical Hernias and common 
surgical procedures. The newer techniques, etc. should 
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32 be covered so that the G.P. can have some idea as to 
cont. which specialists (.consultant) is doing the best for 

his patient on the specific referral. 

33 The visual area can be improved especially when topics 
appear over light background so it is difficult to 
read. The project was basically good. Nov; that it will 
be over it would be useful to have a projector to re- 
view filmstrips unless these too have to be returned to 
you. 

I have some of my projects at: 

Medical assistants monthly meetings 
Hospital - for IPPB (inhalation therapy dept.) 

- for newborn care - and Eli babies . 

These programs could be used by hospital in educating 
or orienting new nursing staff. 

34 Presentation of programs that relate to each physician's 

practice, as no GP works in all fields. 

36 1. Better selection of subjects. 

37 Confine the project to practical applications to improve 
the standard of general practice. 

38 I honestly can't think of any better system. 

39 I believe a regular filmstrip with record or tape would 

be better so you can go back and forth. It would also 
be much less cumbersome. I used these in giving courses 
to interns, but moving the machine each time proved to 
be a handicap. I would also have a very brief printed 
outline of the material with each subject. Otherwise 

it was an excellent course. 



40 



41 



1. Need repair instructions kit. The record and the 
slides frequently do not jive. Trips to the repair 
shop takes two to three weeks in Denver . 

Except for review, where it is helpful in stimulating 
recall of a wider background in a shorter time, and 
where it can hold attention despite being repetition 
of known knowledge, it would need to be presented in 
greater detail and depth to be a very sound background 
for new knowledge and programmed instruction may be 
more efficient here. 

I don't know that I. have any. IT is an interesting 
media; probably more amenable to group instruction than 
to individual instruction to very many individuals. 
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44 1. Director efforts toward the practical use of new 

information. 



46 1. If possible continue to improve least useful aspects 

listed. 

2. Avoid the commonplace or obviously well known and 
avoid the very unusual or complicated subjects which 
might be of interest to only a few specialists. 

3. Additional programs on dermatology, gyn - cardio 
vascular - causes of eczema, etc. would be of gen- 
eral interest and valuable to all. 

(The program on hematology of nev; born was excellent.) 

4. I felt the Medical Information Project has been well 
worked out and programs generally were excellent. 

I can note no further suggestion for improvement. 



47 I like the system. 

For the most part it moves too fast for good comprehen- 
sion and retention with the equipment operating in the 
standard manner. This, of course, is overcome by 
stopping the filmstrip manually until its contents had 
been digested. 

I find summaries quite helpful. Perhaps a summary would 
be helpful for other students. 

Having the program available for review, is a big 
advantage . 

Occasionally I have wondered if the programmer has been 
too much research oriented. I'm sure the programs would 
be presented best by. the man who has held practical exper- 
ience . 

48 None. 

4 9 Enlarge Screen . 

Adequate Reference Outline. 

More emphasis on principles of treatment. 

If this is set up in hospital library it could be very 
helpful . 

50 I don't believe it could be improved much. I would like 
: to see it become a regular activity of yours and other 

institutions. Perhaps it could be made available on a 
free enterprise basis. I would be glad to pay a reason- 
able fee for the program. 

51 To teach in greater quality of material or information. 
Somehow I left each study with the idea that we had be- 
gun a lesson and only gotten to the freshman year of the 
course . 
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52 Disregarding the question of cost (and this seldom seems 
to be of concern these days) a larger sized screen - to 
enable small group participation. 

A projection type onto screen - still with some audio 
correlation and "tests" available for each of group. 

I'm thinking mostly about use in hospital staff groups, 
study groups, nursing groups, etc. 

I'm certain the educational aspects of this approach 
are sneaky, sly and get psychologically effective. 

54 Better visual method. 

55 I felt that the project was well presented. I v/ould 
subscribe to such a program were it available in this 
form. 

56 As previously stated in the preceding paragraph. 

And in giving the presentation more specific drug 
therapy - dosage - etc. for specific conditions that 
arise. Less use of the generic drug terms or give both 
names of drugs. 

57 More length and detail to program. 

58 Smaller topics and discuss in detail more frequent use 
of trade name drugs. 

Concentration on fields where the visual aspects are 
important such as skin lesions tumor, etc. 

Picture of machines (respirator, etc.) are of little 
or no value while the difference in principle would give 
me a better understanding. 

60 To be most practical there should be a completed course 
of programs - which could be used like a lending library 
and also a method of having a dialog between practitioner 
and source of knowledge. 

61 Presenting material that is new, controversial and/or 
practical . 

63 Punctuality- 

Method of selecting parts of records. 

Faster moving. 

Fewer slides used as window dressing, i.e. bottle of 
wine sitting on table. 

64 Have no helpful suggestions. 

66 Improvement in photography or reproduction thereof is 
about my only criticism. 
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No. 

67 Treatment in greater depth of some subjects and not 
' attempt to cover too wide a field in one program. 

A more sophisticated approach with emphasis on infor- 
mation presented a higher level of professional shill 
than was done in some of the programs. 

Technical quality of the program was outstanding and 
little could be done to improve this. 

Most programs should be reviewed at least once to obtain 
maximum retention of material presented. 

73 More specific information; more stress on methods of 
treatment . 



74 Have a sheet at the end of the program - looseleaf type 
that would fit into a pocket notebook with short 
summary of program. Soma of the photography was not 
as clear as it could be (poor contrast on the film). 
Regular spacing of sending out the programs would have 
been appreciated; at times some came two weeks apart - 
others over a month. I sincerely wish to thank you for 
being asked to participate in this wonderful educational 
program. I’m sorry that there wasn’t a course each 2 
weeks during the year-perhaps it would be wise to have 
it 9 months - have during the summer month's. 



77 


1 . 


More depth to discussions. 




2. 


Some t r e atme n t s . 




3. 


Longer discussions. 


78 


None . 


79 


1 . 


Improve items on #12 . 




' 2. 


Reduce mechanics if possible 




3. 


Make viewer more dependable. 



80 More programs per month - one per week.. 

Being in general practice I would like to see a continued 
variety of programs. We are out in country where other 
type programs are not available. Including education 
T.V. programs. So our source of media is monthly maga- 
zines - tapes and this visual program. May it become 
part of the life of the general practitioner. I have 
enjoyed very much participating in this very outstand- 
ing program. 



Use film strips of greater length, 
of records. Improve photography. 



Use tapes instead 







81 



Doctor selects programs he wished to receive. 

More detailed quiz on content. 

Improved AV system 

e.g. 12" recording or roll film strip. 

Maintain library of programs. 

Rent-lease or sell equipment. 

1. Offering a general practice audiovisual course with 
a list, of diverse program subjects allowing the par- 
ticipant to select from these subjects the programs 
he felt most interested and/or need in. 

2. Presenting the entire program on one record (or auto- 
matic record changer) and one tape avoiding inter- 
ruption of program. 

3. Presenting as last film of "one chart, pictorial 
and/or word, demonstrating the salient points of 
the entire program. 

Broadening its subject matter, i.e. course on Ecg's, etc 

1. Record or tape which could move forward or back. 

2. An index of subjects to be presented. 

3. You asked us some good questions but we sent them 
back - it would be very helpful to have many more 
questions at the end of the lecture - this would 
reinforce the memorizing of material much better. 

4. Source material for each subject should be sent as 
a reference - also helps us in questioning the 
author . 

5. If the next program could be sent on receipt of the 
old one. This would help one to cover the entire 
program* at one's own speed, e.g. twice weekly or 
twice monthly. 

6. If costs are reasonable I predict success for MIP. 

I really can't think of any recommendations for improv- 
ing the MIP system - I enjoyed the subject material - 
it was educational and diversified enough to be quite 
helpful in my practice. 

Technically none - continued care to present useful 
information. 

1. The machine itself requires too much maintenance - 
my particular machine required servicing twice (one 
time for nearly 3 months.) 

2. The machine should be able to handle all four strips 
and record without requiring changing. 

The subject material was probably not sophisticated 
enough - a little too basic . 
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90 Stop providing basic information and deal with more 
difficult aspects of subjects. 

91 Make available an inexpensive filmstrip projector, as I 
suspect the present machine would be quite expensive. 

92 None written. 

94 More direct review of instinct of physician. Somewhat 
less pedantic approach to the material - most doctors 
aren't too slow to learn (I hope). 

Better color to the slides would have helped - some of 
them were difficult to make out. 

95 1. Less frequent changing the records and the strips. 

Make a larger record and use a longer film, such as 
a flexible film strip. 

2. I would like a quick means of finding a specific 
part of the program without going through the whole 
record - like the old reel tape recorder, fast, 
forward and rewind? or a short return like on a 
dictation machine. 

3. Identify source of information and/or give various 
viewpoints . 

I have enjoyed participating. I have in mind "talking 
books" available to more people than "legally" blind-on 
records or on tape. Radio stations should present LOTS 
more reading of books and stories . 

Medically-radio and TV medical programs are good in this 
favored area. 

96 They could be made longer and more in detailed. 

The program has been instructive and beneficial, and 
thoroughly enjoyed. I am very grateful for the course. 

97 I feel that TV tapes offer the best form of teaching 
available at this time. These can be used at home for 
a modest investment and during one's free time. Audio 
tapes in the car are very useful. 

No recommendation - all in all enjoyed program, and 
think it would be of value to about any physician. 




99 
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SELECTED COMMENTS 



PROGRAM 1: CARDIOPULMONARY RESUSCITATION 



Positive 



For my convenience it is a good system. I can utilize this method at my 
convenience, especially in the early morning when I am not tired cind in a 
more receptive state to learn. Also I can go back and review as desired. 

I feel I have definitely been benefited by it. This type of emergency occurs 
more than once a year in a busy general practice . I expect to face the next 
episode with more confidence than otherwise might be the case , since I 
fee) that my information is authoritative and current. 

Excellent teaching vehicle due to clarity, brevity and organization. 

Cardiopulmonary resuscitation was well presented. Whoever put this 
package together did an excellent Job of editing out a lot of excess mate- 
rial which would have weakened the impact of the presentation. 

I was favorably impressed with the format, the information dispensed and 
the mechanics of the presentation. 

Much of the information I knew. Some of the vital points of cardiopulmonary 
resuscitation I have not known. This program has organized my knowledge of 
the subject. The pictures are clear in mind as on the film — shouldn't be 
hard to illuminate in an emergency situation. Also — good fun! 

This program is excellent. It taught some important information which I 
need in my practice. It corrected some erroneous impressions I had re- 
garding CPR. 

Have been thinking of keeping a small hand respirator in my house-call 
bag . I will now get one ! 

Rapid, clear, unredundant presentation of important material. Available 
at my convenience, and with ease of application. 

I personally felt a need for this program. I felt it was logically and con- 
cisely presented, well illustrated and did not leave a lot of unanswered 
questions. There was only a minimum of extraneous material. 

Well covered subject of a very important subject. Basic principles well 
pointed out. Would be an excellent program to present to all medical 
personnel faced with this problem . 




SELECTED COMMENTS 
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PROGRAM 1: CARDIOPULMONARY RESUSCITATION 



Negative 

Felt the visual product could have been better with more emphasis on 
technique s-ri. e. , positioning, Jaw control, etc. and not so many 
pictures of apparatus . 

I don't believe it would be as good as a medical film because the con- 
tinuity and flow of information is interrupted by pauses for change of 
slides . 

Good program for first aid class, nurses training, ambulance drivers, 
etc. Necessary information for physicians could have been put on one 
filmstrip and one side of record. Program much too simple and basic for 
physician level. 

At level of Boy Scout training. Excellent for them. Good presentation 
and valuable information. In 20 years I recall the need for this only once. 

It took too long to present a small amount of information — otherwise 
reasonably well presented. 

Material presented at a very elementary level, actually at verbal level 
of laymen unversed in medical terms. Program would be suitable for in- 
struction of anyone interested in basic first aid techniques prior to hospi- 
talization. In this sense, I believe this program to be of a sub-profes- 
sional level of technical information. 

Program was excellent, but the topic in my particular case was one I 
already had read about, seen a film about, and attended a workshop 
about, so it was nothing new. 

Program was elementary and I have almost constant exposure in emer- 
gency room of hospital. However, information was clear and under- 
standable . 

Over- worked subject. Inability to easily refer back is a handicap. 

The presentation was excellent, but for a practicing doctor I feel it 
was too basic. Additional information such as monitors, IV support, etc. 
would add greatly to this program . 

I was well aware of all the information, through presenting the movie 
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film on same procedures to first aid groups, but this audio covered it 
equally well, but not as well visually. It was excellent. 



This program was useful — I had ideas reinforced and picked up new 
aids. It gives a systematic approach to the subject which often times 
is quite aggravating to the physician as well as the patient. I think 
the program is excellent because I can put it to use, probably today, 
and my patients will be getting better care . 

Very practical. I see cases of vaginitis every day. We make a hang- 
ing drop and can usually identify trichomonas and often monilias. If 
monilias is suspected and not found, we do a saline culture at room 
temperature and recheck at 24 and 48 hours. However, we have not 
been diagnosing hemophilus, so this program is of value. We plan to 
do gram slides when necessary. 

I like it because, as a refresher, it showed me what I was doing in 
treatment was correct most of the time; but also that since I last had 
any formal instruction, certain changes in therapy are now recommended 
based on a classification of vaginitis to include an additional type 
(Hemophilus) . 

I felt that it was well presented and helpful and appreciated the use of 
generic and trade names which are often lacking but very helpful to an 
isolated GP. 

Excellent program — very useful in general practice. Had a couple of 
new points, e.g. , in mixed information to test husband, also with 
tetraeyeline . 

I received this program with marked enthusiasm. The information pre- 
sented was a well organized approach to a subject which until now had 
never been presented as an overall subject. The relationships of the 
various forms of vaginitis was never as clearly presented to me before 
as in this program. I am sure I can do a better job of diagnosis and 
treatment as a result of this presenta? on. 

Good presentation — brings out the important facts without extraneous 
trivia . 

This program presented one of the basic problems in general practice in 
a clear and concise manner. Much of the information was already known, 



PROGRAM 2: VAGINITIS 



Positive 
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but it made thinking clearer and more organized . 



Excellent! It was well organized, relatively brief review of the sub- 
ject. I am liable to become enthusiastically in favor of this audio 
visual plus writing plus examination technique of teaching. 



Presented a review of what I consider very basic knowledge — I feel my 
intelligence was insulted a bit. 

The information was generally well known information — very little new 
was learned and only a small aspect was considered likely to be of any 
future value . 

A good program. Not very original or new information, but a very good 
review. I, for one, am interested in a new and challenging program. 

Good presentation, but material again rather simple and basic. I can 
truthfully say that I gained no information that I don't already know. 

Good program and well presented; however, the information was rather 
elementary and already possessed, I would suspect, by most good G.P’s. 

Covers the subject well audibly and visually. However, this is a sub- 
ject usually quite familiar to anyone doing general practice or gynecology. 
A good review although not new or different. 

Information clearly and accurately given; some gaps in knowledge of 
sources, epidemiology, psychologic and psychiatric and other problems 
related to these diseases could be dealt with profit. 

The subject matter was fairly well known to me, but was a good review 
and there were one or two new ideas derived. The presentation was very 
good with the possible exception of several mispronunciations by the 
narrator which makes me feel he must be a professional announcer but not 
a physician. 

Again audio excellent, but the visual — ugh! Poor photographs, only 
slight resemblance to true clinical picture. Material in visual redundant, 
repetitive, poorly thought out (or none ?). Obviously done by a non- 
medical technologist, thus of no interest to me. After 20 years we learned 
to shun this type of horse manure . It taught me nothing ! The visual 
drags and if you are ever going to put this method into general use, you 
must improve the visual. You won't have many takers with the present. 



Negative 
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SELECTED COMMENTS 



PROGRAM 3: T. RUBRUM 



Positive 



This program, being more detailed than the first two, presented a use- 
ful subject in an interesting manner. After running it through the pro- 
jector twice I felt well-informed on the subject and compliment you on 
your effort. 

This program was well chosen because of the neglect clinicians have for 
the condition. Perhaps I question its practicality because of my own 
failure to get patients to take long term therapy for a nuisance problem. 
Since it was well presented, I will be stimulated to attack the problem 
anew and see if I can't persuade some patients to get rid of these ugly 
nails and scaly lesions . 

Forced me to reconsider, re-evaluate and refresh myself on subject which 
I have too often neglected, ignored or summarily referred when perhaps I 
could have taken care of the situation, saving the patient the necessity 
of fairly long trip for consultation with a dermatologist . 

Clinical differentiation will still be difficult in my hands., especially 
borderline cases. However, the KOH preparations should be a help in 
differential diagnosis . 

This program was well presented and provided useful information, some 
of which was new or was a clarification of material with which I was 
familiar only in rather hazy terms. 

I like this program. A very well discussed subject that GP's are con- 
cerned with. Well presented. Important information that I may have known 
but had forgotten about. 

The content was very good and presentation was very good. This subject 
is too frequently skipped over lightly and missed by many of us. 

Excellent presentation. As a GP I found this presentation extremely 
useful in day to day facts . 

Very good. It took one aspect of dermatitis and covered it as thoroughly 
as necessary for my needs in general practice. 

Perhaps skin is of secondary interest, but after viewing the program, I 
found it excellent and with considerable information. 




Excellent program; well presented; easily understood. 
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Negatlve 

Too much time on too little Information. 

The program was too long and time-consuming for a topic of such 
minimal importance . All the information was known prior to this pro- 
gram and if not, it would not have been so bad. 

A dull subject — well presented. A rather poor attention-getter. Compar- 
able to classifying the cracks in a sidewalk. 

Overall quality was good, but visuals left something to be desired in 
terms of clarity and sharpness in some skin slides. 

The program was not technically as good as prior programs — chiefly 
because of poor illustrations. Also feel verbal continuity was confus- 
ing. 

This program was well done , but due to the very small percentage of my 
practive to which it is applicable it was not worth the time. 

Required about 45 minutes to set it up. Could read in illustrated form 
the same information in about 10 minutes. The voice is clear but too 
mechanical — tends to cause my concentration to wander. 

Very helpful review of this portion cf dermatology. My orly criticism 
is the poor quality of visual reproduction . 

An interesting program. Some of the photography of the milder cases 
did not come through well . It would be helpful if the machine could be 
stopped and backed up a little rather than having to start at the first 
of the record or slide — interruptions, e.g. phone calls, make it nec- 
essary to rerun often. 

The photography is poor, especially for dermatological conditions. Sub- 
ject matter is good and logically presented. Continuity is somewhat 
amiss, and I maintain that a small movie projector with accompanying 
sound film would be superior and probably easier to mail. 

Since so many other related skin diseases are mentioned — more treat- 
ment of these other diseases should casually be mentioned also. 
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SELECTED COMMENTS 



PROGRAM 4: FAMILY PLANNING 



Positive 



This represents a timely topic. The Information presented will greatly 
help in explanations to patients. The quality of my prescribing will im- 
prove . 

Though I was familiar with most of the information in this program it 
clarified or corrected my knowledge on a few points (e.g. , relative fre- 
quency of uterine perforation with insertion of different types of IUD's). 

It was well -presented. I would like to have had more information in- 
cluded as to choice of relatively higher or lower progesterone -vs. - 
estrogen content in relation to side effects of oral agents. 

A good practical review of statistics and methods of contraception. It 
was concise and practical, especially in relation to hazards involved 
in some methods and adverse effects so important in the selection of 
these for some patients. 

This program is worthwhile because it emphasizes the importance of 
the problem to the couple hoping to practice family planning and the more 
detailed role of the family physician in helping the couple with their 
plan. This also furnished me with some information which I accept as 
authoritative as opposed to what a detailman may say. We had a 29- 
year old patient on pills suffer a CVA. 

A timely subject as this subject occupies approximately 1/4 of my prac- 
tice. It reassured me on various methods of control, effectiveness, 
side effects, harmlessness, etc. 

I thought this program was a good review of all the modalities of contra- 
ception. It presented some new material to me on IUDs. In general, it 
was informative time well spent. 

Clear, short, concise explanation. Good illustrations; slides and voice 
easy to correlate and follow. Excellent material in comparison of method 
in short, easy to follow lesson. 

I thought this program brought together several methods of birth control 
and evaluated them briefly and yet adequately. 

Again very practical. Uses all modalities of learning — graphs, demon- 
strations, and is frank. 




APPENDIX VIII 
- 8 - 

Good topic. One that bears regular review. Material seemed up to 
j date. Illustrations easy to follow. 
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Negative 

The information was well presented. Most of it was already known and 
subsequently lacked challenge. 

Information of almost common knowledge among professional and lay 
people . 

Repetition of too much well known facts. 

Generally good. Information generally quite well known and experience 
in the subject led me to some disagreement between myself and the sub- 
ject content presented. 

It was a complete waste of time — all the information was already known. 
Had I seen this for the first time on the subject I believe I would have 
rated it "very good . " 

Interesting, but already well known by most physicians. > 

Program good as to informational content. The presentation was too 
rapid, and for me, difficult to follow and digest all of the material as 
presented because of this rapid presentation. 

Again, it contains information that is common knowledge to most physicians 
and, therefore, not very informative or interesting. 

The pace of the program seemed to drag — perhaps this has to do with 
the need to change strips and records . 

The information was quite well known to me . The graphs in the audio- 
visual presentation were not very useful as it was not possible to study 
them without losing out on what the speaker was presenting. Program 
was not very useful to me . 

Not too valuable because it covers information in which I was quite well- 
versed . 

It contains information that is common knowledge to most physicians 
and therefore not very informative or interesting. 

The type of contraceptive most commonly used now is "The Pill." An 
entire program should be used just on that subject. The rhythm, condom, 
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foam , jelly and diaphragm are well-known methods and too much time 
was spent on that subject. 

I feel that this program contained excessive quantitative figures for 
practical use . 



Excellent presentation — difficult to improve. I have no criticism. Ex- 
cellent program as academic or as a review. 

It gave me new information regarding exams, and pleased me by con- 
firming some of my present procedures. 

I thought this program in general was an excellent review of the things 
to be considered in a gynecological exam. The material was well 
organized and the illustrations and photos were generally good with 
the exception of a few of the color slides. 

I think it is an excellent program — a good review for breast and pelvic 
exam — early detection of cancer in either area can be of considerable 
help to patient. 

The pelvic exam as shown here is an essential part of a physical. The 
breast exam technique was clearly illustrated. Both of these exams 
were well demonstrated in this program . 

Very good refresher-review, and good informative teaching aid to office 
sides. 

The program is good. It. took you from the simple facts to a more detailed 
area and also added some things that one does not usually think of. 

This is a fine program. It reinforces and renews information for a complete 
gyn. exam. It also presents some information I heretofore had not used, 
but will incorporate in my examinations . 

So good I'm having my office assistants see it so as to better under- 
stand (and answer patient questions voiced to them) the purposes of 
each step of the exam. 

Program quite helpful. It proved that my exam is incomplete . I have not 
been examining the pendant breast. I rarely aspirate cysts. I have 
not been particularly concerned with outlet tone and sexual function. 



PROGRAM 5: ROUTINE GYNECOLOGICAL EXAMINATION 



Positive 




/ 
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Gave an excellent review of breast exam, and also reminds us to con- 
tinually include pelvic exams in our physical exams and to do routine 
pap smears at least once a year. 

I felt the program was very informative and interesting as the female 
exam constitutes about 60% of my practice. 

Considering time involved, I would say this is very profitable. 



Good for medical school . Waste of time for practicing physicians who 
have done hundreds of pelvic exams . 

Subject well presented. Material is well-known; not particularly useful 
to me. 

Well-presented and very clear, but information too commonly known and 
used in daily practice to be that valuable. 

I think your time and effort can be better expended on more useful topics. 

Good program, but not geared for us GPs. Again, for the beginning 
practitioner or intern and resident. All this material I was very well 
acquainted with and aware of. 

Poorest of programs presented so far because it serves no need. Such 
information must already be thoroughly mastered to be a competent 
physician. 

This would be suitable for medical students . 

Important subject, but it did not present new information of use to me. 

I felt it was somewhat disjointed and did not cover the subject completely. 
Maybe it should have been longer. 

Nothing new or of any value in everyday individual practice . 

This would be OK for a 3rd year medical student, but most of it is old 
hat to those of us out in practice . 

I did not feel the photography was very clear. Also felt the subject was 
not gone into with very much detail. It was OK as a review, but nothing 
new was developed. 



Negative 
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Good basic review of procedures that were taught in medical school. Sub- 
ject is important and I suppose review of it is in order, but I think the time 
could have been used for more valuable subjects. 

Not new — material was learned in medical school and practiced since. 

Some of the slides are of very poor quality, photographically, and, there- 
fore, poor for diagnosing pathology. 



The 3 classifications of small infants was a revelation to me. The facts 
were stated rapidly and I will have to review the program to fully assimi- 
late the content. Excellent subject matter! 

As 'stated, the information here was new (largely) as I am ill-informed in 
neonatal care. Excellent program — well presented. Very informative for 
me — though perhaps very basic to others. 

I was greatly impressed. Much material was presented. Several areas 
of emphasis were given to aid greatly in patient evaluation. This is 
easily the best presentation to date ! It gives more information and presents 
it well . 

Outstanding program in a very important and forgotten unknown field. In 
general practice we are confronted with newborn problems constantly — 
this is a great aid ! 

One of the better programs. As a GP in a small town, I do not see a great 
many prematures. This is a good review of information and therefore use- 
ful. 

Exceptionally good in providing me with new information. I have felt 
a need to be up-dated, but never seemed to find the time for textbook 
study. This is just what I have been needing! 

In my practice, I must evaluate new-boms 50-60 times yearly. The pro- 
gram presented new and very useful information to me, in a manner which 
should make it easy to remember. 

This program was good and timely because it dealt with a problem which 
is frequently faced but for which we are given little information from 
other sources. 



PROGRAM 6: ASSESSMENT OF MATURITY 
AND ENVIRONMENT OF THE NEWBORN 



Positive 
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I thought this program considerably above the level of some previous 
ones in regard to content. Most of the material was new to me and of 
considerable value in my practice. 

As a GP doing OB, I have to take care of newborn, be it premature, dys- 
mature or undergrown. This program reviewed very well these problems 
and "refreshed” my experience. 

An excellent program because it is audio, visual, requires participation 
mentally and actively and tests to evaluate results. 

Negative 

Too brief. What was presented was clear and concise . 

Program was well presented. However, fora GP the usefulness of the 
information may perhaps be forgotten as he is now gradually seeing less 
and less of deliveries and thus less premature-dysmature infants; but it 
gives us a good review. 

Tried to cover too many details in too short of a program. 

Very good, but since I don't treat such cases, I don't have much use for 
it other than as general knowledge of medicine . Am in group practice 
and the pediatricians take care of such cases. 

I found it hard to concentrate and separate the classifications of the three. 

Non-pertinent since obstetrical and newborn post-hospital care is minimal 
in my practice . 

Program developed first part too rapidly with no time to digest material, 
espf statistics . 

To be of more benefit in this area would be to give some specifics in how 
other specialists handled feeding and care of infant. The diagnosis and 
classification was good, but the most important is the feeding and 
environmental aspects of care. 

The photography was quite good. The audio portion was not long enough 
and did not go into sufficient detail on the subject . Only the high points 
were hit. 

New material, well-presented — wished for more detail and additional 
material . 
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I think this program was well-presented and would be of great value to 
pediatricians and obstetricians. I do not have occasion to use the 
material presented so was not initially interested in the topic. 

All new information. Sometimes, not enough time to absorb information 
from charts. 



Positive 



PROGRAM 7: SKIN TUMORS 



Excellent program of vital importance to any physician. I appreciated 
the thought to treat the "whole person" in dealing with skin malignancies . 

Like program — good practical problems we meet each day. Very worthy 
program . 

Excellent choice of subject matter; well presented in almost ideal con- 
ditions . Very applicable in private practice . 

The information keeps the physician posted on new developments and cer- 
tain reviews for us the importance of this phase of medicine. 

I think it is of great practical value, easy to understand and reasonably 
through with no excess "chaff", no fillers. 

Some new information for me and an excellent review of the remainder. 
Simply and precisely presented, factual and not speculative, but easily 
holding the viewer's attention. 

It was a reminder to the G.P. of the needs and importance of biopsy of 
skin lesions- -suspected of being malignant. Good audio;. good photos. 

A good program — helped me to classify and diagnose better. The basal 
cell and squamous cell differentiation was particularly clear in visual and 
lucid in audio * 

This is a well planned and presented program. The entire program is a 
valuable one in general practice. The photography and presentation 
are excellent. 

I thought this was an excellent review of the various skin tumors 'with an 
additional review of the latest thinking on treatment. 

This program covered well the "high points", the important aspects, of 
office -encountered skin lesions — stressing important points well. 
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A useful program well presented visually and orally. Very "meaty", no 
wasted words. Required more time to complete than others because of 
the amount of material . 

This was a good program because it adequately met a need. 



Attempted to cover too much material in time allotted which allowed only a 
superficial treatment of subject material . Possibly an in depth discussion 
of limited topics might be of more lasting value . 

Good quality program. Not a new subject. Most GP's should know this 
material well . 

Good quality color photography with good sound except for pronunciation 
which in some words was different than I had known. 

More emphasis on the early appearance of melanomas and differentiation 
of benign nevi would have been helpful. I am sure many of these malignant 
lesions are perhaps unnecessarily and wrongly punch biopsied. 

The topic was excellent. More detail could have been used for methods 
and techniques of biopsy. 

For a student — I would assume this a good program. For a GP, nothing 
new and time-consuming information. 

Format was good with comparison teaching put forth and satisfactorily 
described. Needed some additional basic surgical approach to these 
lesions if could be included. 

For skin lesions there are obvious reasons why a photographic presenta- 
tion is superior. A greater variety or more examples of the skin lesions 
would have been helpful . 

I was bothered by apparent lack of supervision in word pronunciation. Pro- 
gram organization was good. Visual program is only effective teaching 
method to demonstrate skin lesions. Program format could extend to 
differential diagnosis . 

A good program for one beginning practice. However, with experience, by 
actually dealing with these conditions, one gets reassurance that he is 
doing as could be expected. 

Information cleared up several points for me . Time-consuming for informa- 
tion obtained . 



Negative 
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Program was good. Narrator had problem with words "keratosis" and 
"seborrheic. " 



Although I do not do deliveries, there is always more to learn and you 
never can tell when an emergency delivery may come up I 

Practical topic. Largely review. Some excellent points. 

Excellent program. Reviewed the obstetrical problems that face us fre- 
quently and need to be handled correctly. A refresher course in problems 
we should all be acquainted with and know how to handle . 

This was an excellent review. It reviews the more common delivery 
problems in obstetrics. 

Doing obstetrical work in a rural area, this program is excellent. It is 
also a much needed review, as I have not had a good review at any of the 
courses since I left medical school 15 years ago. . 

Well -organized , well-edited, well presented. 

Excellent program. Very important information presented in an interesting 
and easily understood manner. 

Program was very well organized, dealt with the high points, etc., 

especially when you consider some of the variable circumstances in 
obstetrics . 

Excellent refresher for practical diagnosis and treatment. 

A good program dealing with a subject that needs clear thinking on part of 
physician when these situations arise. 

The material was excellent for use by any GP doing obstetrics and was a 
good review of the material. 

Excellent— clear and much excellent information I did not know before . 
Especially liked the drawings . 

It clarified my thinking on these problems., so that I should be able to approach 
similar situations with more confidence . 



PROGRAM 8: OB EMERGENCIES 



Positive 
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Thls program was so full of concentrated, important guidelines for the 
management of obstetrical emergencies that digestions and assimila- 
tion was difficult without repeated "re-runs" ! ! 



It was an excellent review of a subject which is becoming foreign to me. 

Not very good for my needs as I do not do obstetrics . 

It would be very good if I were doing OB. I am quite out of practice in 
OB but this program was good to refresh my pervious training and limited 
practice — just for information. However, for my needs it was brief. 

Is helpful information and nice to be reminded of this, but probably 80-90% 
was already known to me . 

I would prefer a more detailed coverage of this program which covers the 
subject more in depth. 

Too basic. Program OK for 2nd year medical students. 

Again, too basic and would only be of interest to someone who does not 
practice OB. If you do OB work and don't know this material, you and 
your patients are in trouble. 

No new information. No aid in treatment. 

Since fetal distress is such a difficult determination, I think this program 
should be up-dated, and some hope of better early diagnosis presented. 
Perhaps the newer techniques are still too controversial. 

This type of program would be excellent for medical students and interns . 
However, all practicing physicians doing OB should have this information. 

The review is good for everyone and should make the physician more sure 
of his treatment. 

As stated, of little personal value (no OBs in practice) but of general in- 
terest. 

Excellent review of these OB emergencies, but perhaps too brief. Topic 
should be expanded to two programs . 

A little confusing, should review it several times. It would be better if 
visual was tailored to fit in better with audio and done in same caliber 
as audio. As it is, repugnance is the eventual reaction of one expecting 
a high caliber teaching experience and getting only poor graphs , meaningless , 
unrelated pictures , and downright absurdidites . 



Negative 
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SELECTED COMMENTS 



PROGRAM 9: TAUNDICE IN THE NEWBORN 



Positive 



I regard it very highly. It required close attention because of the compact- 
ness of it. The repetition was right on the nose, making it easier to re- 
tain more of the information. 

To a GP doing OB this program, I think, presented an everpresent problem 
of the newborn — the use of "photo therapy" 1 found very revealing and in- 
teresting. 

The program concisely presented material very important to those of us who 
care for newborns. Although I actually knew the material presented, the 
program brought it into sharp focus , and should improve the quality of 
medicine I practice . I consider my time well spent and intend to study 
the program further. 

For me this program was excellent. The presented material gives me much 
better over all picture of this problem than I have ever previously had as well 
as correcting some misconceptions. 

This program was one of the best of the series. This audiovisual method 
lends itself well to such subjects; giving a student plenty of time to study the 
detailed outlines and graphs. 

Best of series — very good presentation of accumulated knowledge of subject 
presented uncluttered by myths, old concepts, etc. and yet adequately 
presented latest established theory, methods of diagnosis and treatment. 

Best program to date both in content and method presented. The material 
was very well organized and presented and much of it was entirely new to 
me . 

Best program yet. Seemed to be crammed with detailed and pertinent in- 
formation. 

Jaundice in the newborn has been inadequately approached by me. This program 
corrects the inadequacies and will provide me with greater self confidence. 

This program is timely and excellent information needed for anyone doing 
OB and pediatrics. I usually have to review this type of information each 
time I have an acutely jaundiced infant. This reinforces and adds to my 
knowledge on the subject 
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Very good, especially part on phototherapy. 



Negative 



Too much listing of factors and confusion arising from "do's" and "don'ts" 
presented. 

I think the program would be fairly Informative to one who was interested 
In this subject. It was not Interesting to me, as It presented highly 
technical material which I would never use, in an uninteresting manner. 

Program highly condensed, concentrated, and therefore difficult to follow 
without several sessions. 

The program was well-presented and fairly easy to follow. More so than 
others in past; there is still the problem that if any point is not completely 
understood, there is no way to go back to this area on the records— must 
run the entire filmstrip and record through again. 

A good program. However, aside from diagnosis, therapy is usually assigned 
to a pediatrician. 

No pediatric or OB practice in 23 years. I will probably never have occasion 
to use this information. However, all medical information is valuable, in- 
directly, and this is a we 11 -presented program. 

An important topic dealing with information, some of which was foreign to 
me. It was sometimes hard to follow because of the amount of information 
given and difficulty keeping up with both visual and audio simultaneously. 

Important, but parts were not clear and graphs needed more clarification. 

Of more interest to the pediatricians . The conditions are most uncommon 
and most GP's would refer to pediatricians. 

To increase its benefit, would be Letter to lengthem it and add a few more 
detailed ideas. 

Audio and visuals were not synchronized at times. 

The staccato voice, speed and amount of material are just too much. Well- 
prepared . Probably a second or third review would be profitable . 
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SELECTED COMMENTS 



PROGRAM 10: TRANQUILIZERS 



Positive 



Very outstanding, useful and practical. Very well presented as to content, 
usefulness, types of drugs, Indications, side effects, etc. We all need- 
ed this program because of the overwhelming use of these drugs today. 

I'm presenting this program to the nurses and Interns of our three largest 
hospitals. I was very impressed with this program. 

It was timely, clear, easily followed. Helped clear up the confusion 
caused by the drug brochures. 

Presented information which I need because of the extensive use of the 
medications discussed. Also the knowledge of these drugs that is common 
in the lay public makes it important that the physicians have as much facts 
as possible. More information could have been well included. 

An excellent condensation. The subject has become increasingly difficult 
for me, especially with the advent of combination drugs. I value this 
presentation greatly. 

The principles presented in this program were basic and conservative. This 
was to my liking. In my practice I have considerable opportunity to utilize 
this material, and I am better informed in the subject for having viewed it. 

Tremendous as to its presentation, new material concisely presented and 
germain to our age of psycho-therapeutics . 

It confirmed some of my suspicions of diugs and their effectiveness. It gave 
an honest critique of medications commonly used; such a critique is lacking 
in almost any other source of information. 

It fills a pertinent need for stressing important facts in the use of tranquili- 
zers in medical practice. I find the busier I get in the office, I find less 
time to keep up and review important information needed in the practice of 
medicine . 

I have a general type of practice. Many patients have symptoms discuccod 
in this program — to me it was very enlighteneing . 

Subject matter very appropos. Detail given was in the right degree. 

There is such a confusion welter o£ diugs end infcnmatlvu, •and tliis succinctly 
puts it in perspective . 




Negative 
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Too much covered in too short a time . Hart to identify drugs since I know 
them by trade names. 

Not very adaptable to audio-visual technique . 

Fairly good. The statements made are open to some divergence of opinion 
and were not presented in this way. 

The information was too basic. Would be good for medical or nursing students. 

Much material presented in a didactic monotonous manner with presentation 
of long lists of symptoms, etc. Several opinions presented are probably 
just that and represent impressions of author of text and not definite facts 
supported by experimental evidence, especially concerning the use of minor 
tranquilizers . 

I thought it was a timely topic though treated somewhat superficially. 

I objected to the avoidance of use of trade names. The slavish avoidance 
of a trade name is best relegated to university centers. 

I seemed to feel that the author had a strong dislike for any minor "tranquili- 
zers" other than barbiturates. I think all drugs have their use in special 
occasions . 

Material is not new. 

This program failed to clarify and detail "trade name" drugs with generic, 
tranquilizer classification; practical usage seems to demand a clearer 
identification of drugs as tranquilizers, usage and categories to which they 
belong . 

The audio was OK but as noted previously, the visual was a flop. I find that 
without a good text the programs are just about a complete loss as far as 
permanent use in my practice. There is no place to readily refer for details 
such as dosage. They have represented something useful in my education, but 
there is something radically missing in their permanent usefulness in my 
practice of medicine . 

I thought it was good, but visual hard to follow. 

I need some thought organization on this subject. The information presented 
is useful but presented too rapidly for me to get it. 
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SELECTED COMMENTS 



PROGRAM 11: INHALATION THERAPY 



Positive 



I liked it. We use inhalation therapy both in the sick room and the 
physiotherapy clinic. The information here gave me some points I did 
not know, such as related to volume limits, and pressure limited 
ventilators, concentration of gases, and we have been guilty o,. bed- 
side guessing as to need. Also, I shall henceforth be more explicit in 
writing the orders. 

Good program from standpoint of newly utilized equipment for the patient 
with chest problems. 

I think that it provided an interesting and informative, simplified look 
at this aspect of therapy. It is of value to anyone doing general practice . 

The program pointed up my deficiencies in diagnosing the need for inhalation 
therapy. It also was very useful in manifesting principles of treatment. 

I should do a better Job for having seen and heard the program. 

Excellent, clear, concise — filled a vital need in my Knowledge of inhalation 
therapy. I will use this before returning it. 

A new approach. Seldom presented so succinctly. Reviews most important 
facts of therapy. 

I think the program was presented clearly, very understandable and easy 
to follow. 

It is a neglected subject in my practice because I turn this over to a 
specialist. However, l am interested and should be more experienced; this 
program was a good start. 

I think it to be a valuable program . It was very concise about directions 
in a field which has generally been dealt with rather loosely. 

Important subject for all doctors, as too few are well acquainted with this 
type of apparatus . 

This is a good program, especially because of the recent knowledge about 
inhalation therapy. Fortunately at the hospital where I take my patients 
they have a very good department and have had 2 programs at staff meet- 
ings.- New equipment is introduced to staff and uses discussed. This pro- 
gram reinforced many of the things already known. However, it is very re- 
assuring. 




Negative 
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The subject was much too broad to be covered in this one program. 

This program did not hold my attention as much as many of the other MIP 
programs. The technical elements (photography, charts, etc.) were well 
done but the presentation itself was luke-warm. 

Program good but I probably will not use the information in my practice. 

I am not certain of retaining this information. Perhaps it was insufficient 
in depth . 

Too complicated for a GP to digest. 

It was difficult to cover this subject in very much depth by this approach. 
Hospital or class seminar with demonstration would be better. Perhaps 
more filmstrips would improve the presentation. 

Not specific enough or, treatment. 

No new information was presented on equipment presently available in my 
hospital. I was interested in seeing what material would be presented on 
mechanically assisted respiration, equipment for which is just becoming 
locally available, but the information given was too incomplete, with my 
limited present knowledge, to be of any value. 

Seemed to cover too many machines; subject matter too broad to be specific 
and useful. 

Program is superficial. It is too non-specific, i.e. equipment-medications 
and indications for use. 

Basically good, but most of us need more information on Just how to order 
for say: (1) pneumonia, (2) asthmatic bronchitis, (3) variations of chest 
disease. 

Not up to standard of some previous programs as the material was presented 
in superficial way and did not leave me with information I could apply in a 
practical way to specific cases. 

Audio OK although bits of information were thrown out helter-skelter and I 
had to listen over to catch them. The visual was a real distraction with 
its lack of useful information, and I was pleasantly surprised at how the 
caliber of the program rose when I turned off the visual. 
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PROGRAM 12: HYPERTENSION 



Positive 



Very well presented — good subject matter — practical and useful. More on 
treatment would have been of value. This program will be presented to the 
interns in Oct. at our two largest hospitals. I was very pleased with the 
overall subject matter and presentation. 

It served the purpose of arousing enough interest to make one "go to the books" 
to find out more. 

This was an excellent concise review of the diagnosis and categorization of 
types of hypertension. 

A concise explanation of certain basic aspects of an involved and complicated 
problem . 

Excellent because it is a subject that I encounter every day. 

We in GP are frequently confronted to work up routines for a particular 
problem. Routines are time savers for the doctor and money savers for the 
patient. Only last week we discovered a routine for work-up of the hyper- 
tension. This program adds to the effectiveness of our work-up. 

A very common problem seen in general practice. Condenses information 
for diagnosis and treatment in this important area. 

Good program. It hit high points, is practical. 

Excellent program — clear and concise. 

The subject matter and presentation excellent. 

One of the best of the series. 

Concise, brief review of an important tppic. CVA auscultation new to me. 

It was important because of its appropriateness and wider everyday appli- 
cation. 

Useful information and gives good outline for diagnosis of hypertension. 
Concise evaluation of hypertensive patient was covered. 
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Neqatlve 



The subject was too broad for the length of the program only highlights 
could be touched— it v\ould have been better to choose one category of 
the subject of hypertension and cover it thoroughly. 

Very little new information. It needs to be followed with some idea of 
clinical application. 

This program presented material already well known and understood at a 
level of a sophomore or junior medical school level or as would be pre- 
sented to nurses. 

I think this program was excellent with one exception, the organization 
of the material presented tended to jump from one subject to another 
and then back again to the original subject. ;No definite follow-through 
on one subject before starting another . 

Good, but on such a subject more should be said on various areas— con- 
tinuing to elaborate in subsequent programs would be good. 

I felt it was good but did not do much more than scratch the surface of 
this subject. Also, some of the tests recommended and procedures advised 
were gone over rapidly and require review and the machine is limited in 
the* it is not possible to turn back to review. 

This subject has been presented in seminars, GP programmed material and 
many activities in the literature, and thoigh important, I did not feel that 
it presented anything useful to me. The narrator was objectionable. 

Difficult to evaluate as my machine audio is poor, records do not coordinate 
with slides. Last time machine was worked on it took 2-3 weeks to get 
back. Machine failure may be one of the problems with this type program. 
I've had machine problem during 4 programs. 

Subject matter was at medical student level and of no practical use to the 
average practitioner* 

Poor commentary— fair pictures— not particularly useful. 

I thought it was too brief — would also have liked something on treatment. 

Too juvenile . 

Portions too rapid for easy assimilation, e.g. , urinary tests. 




SELECTED COMMENTS 
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PROGRAM 13: ASPIRATION OF THE TOINTS 



Positive 



I see many cases of joint diseases or injury. This program serves as a 
memory refresher. The overall presentation is excellent, the information 
concise and usable with least loss of time. There appears to me to be 
minimal deadwood and with a little better perspective in illustrations it 
is excellent. 

I think this was the best program in the MIP series. It did not attempt to 
cover too much subject material. . .it was concise, directly presented and very 
useful and interesting. 

This topic lends itself well to the audiovisual presentation. The privilege 
of being able to re-run the strip is especially good. A medical film on this 
subject seen recently was good , but moved a little too fast for good com- 
prehension . 

I believe this will be of value to me in my future practice . 

This program is useful because it can improve my techniques of aspirations 
and I will feel more competent in aspirating other joints. It also, through 
the use of the charts, should improve my diagnostic ability. 

This program has been the Lest so far insofar as new and useful material 
is concerned. 

Important information which I can and will use. Very well presented. 

Program covered material well. It was above average in content. Only 
criticism would be to show needle angle on model not on skeleton. Distance 
of photographing the Joint on skeleton made them of little value. Additional 
information about steroid injections seems indicated as part of series. I 
liked the stress on diagnostic testing of fluid. Very well done. 

Excellent-interesting program. Useful information. 

Very helpful and well done. One of your best presentations. 

Presented useful information. Inclusion of the printed tables was very 
helpful . , 

It was an excellent review of the subject in a concise and easily under- 
stood manner. 
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Very practical and useful In treatment and diagnosis. 

Negative 

Good review, but I seldom aspirate a joint. Record stopped too many times — 
a nuisance. 

t 

Very impractical for a generalist. OK for a rheumatologist. 

Much of the information was quite elementary and a great deal of time was 
spent in describing technique of joint aspiration which should be obvious 
to most physicians. Almost most of the fluid tests described for aspirated 
fluid would be very time-consuming in a busy office, and in my opinion 
would not yield information of great value. 

I deal daily with these problems. No new thoughts were advanced. 

Nothing presented that was new or different from techniques used. 

This was a very poorly put- together operation. 

Not very clear. 

This program was the first time the audio synchronized with visual. Per- 
haps it was a very new machine, but although the quality of the visual 
wasn't too much better, at least it seemed to fit into the audio better. By 
the way, the machines have a long way to go. My new machine arrived with 
the lens loose in the box. 

Good, to the point, but not all its demonstrations were of top quality. 

I do not usually do joint aspiration so this was a good review for information I 
should have. It seemed the information was a bit brief, but how can you en- 
large upon "I am lateral to the tip of the middle malleolus" so that it sticks in 
your mind? This is probably better taught by working with someone where 
actual palpation fastens the idea in the mind. 

I think more information could have been presented and more details given. 

Joint aspiration is a "fixed" stereotyped procedure commonly done at the cilice . 
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PROGRAM 14: ANEMIA 



Positive 



Good program, required more attention and stimulated greater interest to 
again study the problem of anemias . 

Very interesting and well presented. Practical and useful in office and hospi- 
tal. The method of diagnosing was made very simple and easy to understand. 
The subject matter was excellent review and not too deep . The manner of form 
of presentation was excellent. 

Vital topic well arranged and condensed. A minimum of microscopy was pre- 
sented and this I appreciate as I haven't seen a blood smear in 28 years and 
may never see another. Also a minimum of lab procedures and mathematics 
aided in holding interest. 

The program afforded me a better understanding of this common problem, and 
should be a definite help to me in diagnosis and treatment. Every man doing 
general medicine would benefit in some degree by seeing and hearing it. 

I consider it excellent because it presented information that I needed and re- 
viewed and clarified important details . 

Program well synthesized and presented. Practicality level was excellent. 

I thought this program was very good in that it acted as a refresher course 
on anemias and also gave more recent information which I was not aware of. 

It took much of the mystery out of recent articles I have been reading. 

Good program. I tend to let the consultant carry the load. I would be 
smarter to know, too. Well-presented. 

New information; concentrated; refreshing; stimulating. 

A clear concise well presented basic outline of the work-up of an anemic 
patient with emphasis on laboratory findings. Best of the program series! 

I thought this program to be excellent. It gave me a new viewpoint in regard 
to my anemic patients. 

I needed the information in this program. Judging from the mistakes made on 
this questionnaire, I need more programs on anemia or blood studies. 




APPENDIX VIII 
-28- 



Excellent program; valuable information; good review of old and new 
material . 

Negative 

Good, but at times I found it difficult to follow. 

Hematology is a difficult and involved subject to be presented in such a short 
format. 

Just fair. Not complete enough with discussion. 

All very good, but it is a complex subject and presenting it in such brief 
form left a lot to be desired. A doubling of this program on this subject 
probably would have been better. At long last you have started giving us 
good stuff. 

Charts and graphs are difficult to absorb and follow. 

Went rather fast from one important point to another. 

Program rated as only fair in that it reviewed fairly basic physiology and 
although useful, did not present new or necessarily information that needs 
frequent review. 

The approach to the problem was excellent. The logic of making conclusions 
was made obvious. The subject was well-handled. The scope of the pro- 
blem as seen in practice was not complete . Did the author run out of time or 
interest? A subject of this type would be of more value to me if handled 
from the point of view of newer diagnostic tests and their evaluation. I liked 
what was here, but it was not enough. 

Too basic. 

Technically well-presented; however, a little too basic. 



Program good value to physicians who see back complaints, pc. 1 culary the 
maligner test. 

I thought this program was excellent in that it presented a logical and 
simple , yet very thorough means of intelligently examing a back. It is t ie 
first time either in medical school or subsequent seminars that I have ever 
received such information in toto . 



PROGRAM 15; EXAMINATION OF THE BACK 



Positive 
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This program presented needed material in a clear, succinct, well-illustrated 
manner. 

The photography was excellent in that the configuration of the back was well 
shown — particularly as regards curvatures, etc. 

Very well-done; examination methods excellent; I have become sloppy in my 
examination. 

Many patients seem to have back-aches and back problems . I had only a vague 
idea as to the correct type of examination and evaluation. 

I enjoyed it and it will continue to benefit me in my practice . 

Many practitioners are overwhelmed by low back pain (58 different causes !) 

This helps in elimination and diagnosis. 

Excellent program; presented clearly informative useful information. 

Very good. A simple demonstration of practical examination often not done 
well by the GP. 

The explanation along with illustrative photography were excellent. 

This was well -described, covered the material thoroughly, but not too 
rapidly. 

Very interesting information. The program points out things I thought I knew, 
but actually did not know. 

I thought it was very good, as it showed the practical applications of diagnostic 
methods . 

I found it interesting and of value to review and clarify the sort of examination 
I do frequently . 

I thought it was one of the best programs of the series. 



Well done; but not useful in my practice . Questionable audiovisual value . 
Information good for student, but too basic for practitioner. 

Fair review; disappointed in level of commentary. 



Negative 
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Excellent presentation, but material not new. Good review. 

Excellent for student demonstrations only. 

Very good, but rather elementary. 

Good review of examination of normal back . Did not deal with diagnosis or 
treatment. 

Specific areas could be pointed out in dealing with differential diagnosis; 
this would continually help physicians become aware of other possibilities. 

Well-illustrated and well-organized. I wish some suggestions for treat- 
ment were given at least in the common case where there is no evidence 
of x-ray pathology — only the "back ache" is present. 

Very good for the time given. Film would take more time, but cover more 
ground . 

Too basic. 

All material presented was good, but I think some principles of treatment 
should accompany. 

Was very nice. My audiovisual machine does not work well, so this was 
very frustrating to give proper attention. 

While the subject is frequently dealt with, I felt only portions of the pro- 
gram were of real use to review. 

This program presented material that I knew well, but it is well organized 
and a good review of principles of diagnosis for low back disorders. 

Not enough detail on test. 
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PROGRAM DEVELOPMENT 

Approximate 

Cost/Proaram 

A. Cost of film and development (in house) 

(R.G.B. , and Garfield Studios) $165.00 

B. Art supplies (H.G. Daniels) $ 60.00 

C. Consultant $350.00 

D. Researcher $150.00 

$725.00 
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Commercial Reproduction: AUDIO 

Approximate 

Cost/Prooram 



A. Narration: George Walsh $ 90. 00 

B. RCA 

Studio time: $35/hr. (2 hr. program) 70.00 

Editing: $20/hr. 20.00 

Tape rsels: $10/ea. (3/program) 30.00 

7" LPM proc.: $ll/ea. (4/program) 44.00 

M transfers: $10/ea. (4/program) 40.00 

Reference lacquers: N/C (2 program) 0.00 

Printing records: $39/record 

(250/program) lQl.QQ 



TOTAL AUDIO: $395 . 00 



Commercial Reproduction: VISUALS 



C. Identi color/ R.G.B. Labs- 

Intemeg film with processing $ 5.00 

D. General Film Labs- 

Film loop printing: $. 1134/foot 

(1950 ft/program) 210.00 

E . Hoffman- 

Encapsulating of filmstrips: $0.65/ea. 
(500/program) 325.00 



TOTAL VISUAL: 



$540.00 



TOTAL COMMERCIAL REPRODUCTION 

COST PER PROGRAM: $935.00 
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Packaging and Distribution 



Approximate 

Cost/Program 



I 

I 
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i 
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A. Bert-Company Enterprises - 

Albums- incl. production and printing 
$0.36/ea. (125/program) $ 45.00 

(total cost- $1124.30 for 3125 albums) 

B. Packaging Sales Company - 

Album mailers without printing: $92/100 

(125 program) 10.00 

(total cost- $306) 

C. Don Figge - 



Return address labels- $1.25/100 1.25 

Program booklets covers: $222/3250 8.. 00 

24 page booklet: $l/ea. (125/program) 125.00 

D. Addressing; packaging; posting 63 . 00 



TOTAL PACKAGING AND DISTRIBUTION 
COST PER PROGRAM: $ 252.25 
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